Jul 20 17:46:01.871115 ip-10-0-135-108 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 17:46:01.871127 ip-10-0-135-108 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 17:46:01.871135 ip-10-0-135-108 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 17:46:01.871618 ip-10-0-135-108 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 17:46:11.980820 ip-10-0-135-108 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 17:46:11.980836 ip-10-0-135-108 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fef133641c0c433a897552b0f074581c -- Jul 20 17:47:51.954934 ip-10-0-135-108 systemd[1]: Starting Kubernetes Kubelet... Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 17:47:52.315138 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.313396 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318127 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318146 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318149 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318154 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318158 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318161 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318166 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318171 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318175 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318179 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318183 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318186 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318189 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318192 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318195 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318198 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318200 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318203 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.321184 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318206 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318209 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318211 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318214 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318216 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318220 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318224 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318227 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318230 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318233 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318235 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318238 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318240 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318244 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318247 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318250 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318253 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318256 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318259 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.324678 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318261 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318264 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318266 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318269 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318271 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318274 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318276 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318279 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318281 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318284 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318287 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318290 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318293 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318295 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318298 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318301 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318303 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318306 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318308 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318311 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.327252 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318314 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318316 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318319 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318322 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318325 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318328 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318331 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318334 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318337 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318340 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318342 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318345 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318347 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318350 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318352 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318355 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318357 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318361 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318364 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318366 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.330959 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318369 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318372 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318374 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318377 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318379 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318382 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318385 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318387 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318390 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318392 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318395 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318398 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318401 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318404 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318407 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318410 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318413 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318415 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318418 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.362008 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318421 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318424 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.318427 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321582 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321594 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321599 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321602 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321605 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321609 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321611 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321615 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321618 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321621 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321642 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321646 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321652 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321657 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321660 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321663 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321667 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.390743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321669 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321672 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321675 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321678 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321681 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321683 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321686 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321689 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321692 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321695 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321697 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321700 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321702 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321705 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321709 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321711 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321715 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321718 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321721 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321724 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.657445 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321727 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.399001 ip-10-0-135-108 systemd[1]: Started Kubernetes Kubelet. Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321730 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321733 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321735 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321738 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321741 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321743 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321746 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321749 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321752 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321754 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321757 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321760 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321762 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321766 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321769 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321771 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321774 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321777 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.672964 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321780 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321782 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321785 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321788 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321790 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321793 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321796 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321800 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321802 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321805 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321807 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321810 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321813 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321815 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321818 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321820 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321823 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321825 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321828 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321830 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.674110 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321833 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321836 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321838 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321841 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321845 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321847 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321850 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321853 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321856 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321858 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321861 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321864 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321867 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321870 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321872 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321875 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321877 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321880 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321882 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321886 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.674902 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321889 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321892 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.321895 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322309 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322315 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322318 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322321 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322324 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322327 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322329 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322332 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322336 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322339 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322343 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322345 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322348 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322351 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322354 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322357 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322360 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.676106 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322362 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322365 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322367 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322370 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322373 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322375 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322378 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322380 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322383 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322386 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322388 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322391 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322394 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322397 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322399 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322402 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322405 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322408 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322410 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322413 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.676938 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322415 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322417 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322420 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322422 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322425 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322427 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322430 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322432 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322435 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322438 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322440 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322443 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322446 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322449 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322452 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322455 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322458 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322461 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322463 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.678130 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322466 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322468 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322471 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322476 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322479 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322481 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322484 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322487 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322490 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322492 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322495 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322498 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322500 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322503 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322505 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322508 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322510 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322513 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322515 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322518 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.678919 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322520 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322523 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322527 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322532 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322536 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322539 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322543 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322546 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322548 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322552 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322554 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322557 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322560 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322563 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322565 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322569 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322572 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322574 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322577 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322579 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.679749 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322581 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322584 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.322589 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322673 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322682 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322692 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322698 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322704 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322707 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322711 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322715 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322718 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322721 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322725 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322728 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322731 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322734 2554 flags.go:64] FLAG: --cgroup-root="" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322737 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322740 2554 flags.go:64] FLAG: --client-ca-file="" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322743 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322746 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322750 2554 flags.go:64] FLAG: --cluster-domain="" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322753 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 17:47:52.680535 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322756 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322760 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322763 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322766 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322772 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322775 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322778 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322781 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322784 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322787 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322790 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322794 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322799 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322802 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322805 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322808 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322811 2554 flags.go:64] FLAG: --enable-server="true" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322814 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322819 2554 flags.go:64] FLAG: --event-burst="100" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322822 2554 flags.go:64] FLAG: --event-qps="50" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322824 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322827 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322830 2554 flags.go:64] FLAG: --eviction-hard="" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322834 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 17:47:52.681355 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322837 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322840 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322843 2554 flags.go:64] FLAG: --eviction-soft="" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322846 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322849 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322852 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322855 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322859 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322864 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322868 2554 flags.go:64] FLAG: --feature-gates="" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322872 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322875 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322880 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322883 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322886 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322890 2554 flags.go:64] FLAG: --help="false" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322893 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-135-108.ec2.internal" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322896 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322899 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322903 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322906 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322910 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322913 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 17:47:52.682166 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322915 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322918 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322921 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322924 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322927 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322929 2554 flags.go:64] FLAG: --kube-reserved="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322932 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322935 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322938 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322941 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322944 2554 flags.go:64] FLAG: --lock-file="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322946 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322949 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322952 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322958 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322961 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322964 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322967 2554 flags.go:64] FLAG: --logging-format="text" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322971 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322974 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322976 2554 flags.go:64] FLAG: --manifest-url="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322981 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322985 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322988 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322992 2554 flags.go:64] FLAG: --max-pods="110" Jul 20 17:47:52.682990 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322995 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.322998 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323001 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323005 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323008 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323011 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323013 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323021 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323024 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323027 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323030 2554 flags.go:64] FLAG: --pod-cidr="" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323033 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323038 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323041 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323044 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323047 2554 flags.go:64] FLAG: --port="10250" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323050 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323053 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ec32b79a6532664" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323056 2554 flags.go:64] FLAG: --qos-reserved="" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323059 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323062 2554 flags.go:64] FLAG: --register-node="true" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323065 2554 flags.go:64] FLAG: --register-with-taints="" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323069 2554 flags.go:64] FLAG: --registry-burst="10" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323072 2554 flags.go:64] FLAG: --registry-qps="5" Jul 20 17:47:52.683825 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323075 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323077 2554 flags.go:64] FLAG: --reserved-memory="" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323081 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323084 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323088 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323091 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323093 2554 flags.go:64] FLAG: --runonce="false" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323096 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323099 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323102 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323105 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323108 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323112 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323115 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323118 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323121 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323123 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323126 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323129 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323132 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323135 2554 flags.go:64] FLAG: --system-cgroups="" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323137 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323143 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323145 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323148 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 17:47:52.684570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323152 2554 flags.go:64] FLAG: --tls-min-version="" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323155 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323157 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323160 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323163 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323166 2554 flags.go:64] FLAG: --v="2" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323175 2554 flags.go:64] FLAG: --version="false" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323180 2554 flags.go:64] FLAG: --vmodule="" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323183 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.323187 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323298 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323303 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323306 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323309 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323312 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323315 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323318 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323321 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323324 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323328 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323330 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323333 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.685330 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323336 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323338 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323341 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323344 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323346 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323350 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323354 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323357 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323359 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323362 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323364 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323367 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323369 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323372 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323375 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323377 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323379 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323382 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323385 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323398 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.686145 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323400 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323405 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323408 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323412 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323416 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323420 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323424 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323428 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323432 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323436 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323439 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323441 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323444 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323447 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323449 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323452 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323455 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323458 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323462 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.686949 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323465 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323470 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323473 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323476 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323479 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323482 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323485 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323487 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323490 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323493 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323495 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323498 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323501 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323509 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323513 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323516 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323518 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323521 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323523 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.687774 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323526 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323528 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323531 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323535 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323537 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323540 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323542 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323545 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323547 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323550 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323552 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323555 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323557 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323560 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323562 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323565 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323568 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323570 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323573 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323575 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.688574 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323578 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323580 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323583 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323585 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323588 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323590 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323593 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323597 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.323600 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.324227 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.332947 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.332963 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.332983 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.332991 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333047 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.692605 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333052 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333055 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333058 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333061 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333063 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333066 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333069 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333071 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333074 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333076 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333087 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333090 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333093 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333096 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333099 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333101 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333104 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333107 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333110 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333112 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.693431 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333115 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333118 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333120 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333123 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333125 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333128 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333130 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333133 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333137 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333141 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333144 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333146 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333149 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333152 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333154 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333157 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333159 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333162 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333164 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333167 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.695050 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333169 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333172 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333175 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333177 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333180 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333182 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333185 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333188 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333191 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333193 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333196 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333199 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333203 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333206 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333211 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333215 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333218 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333222 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333227 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333230 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.695888 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333233 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333237 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333241 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333245 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333248 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333250 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333253 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333256 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333259 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333261 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333263 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333266 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333269 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333271 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333274 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333277 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333279 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333282 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333285 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333288 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.697107 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333291 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333293 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333296 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333298 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333301 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333304 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333307 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333309 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333312 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333314 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333317 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333319 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333323 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333325 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333328 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333331 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333333 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.697931 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333336 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.333341 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333453 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333469 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333472 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333475 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333478 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333481 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333484 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333486 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333489 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333491 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333494 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333496 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333499 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 17:47:52.801727 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333502 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333504 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333507 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333509 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333512 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333514 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333517 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333519 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333522 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333524 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333527 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333529 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333532 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333534 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333537 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333540 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333542 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333544 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333547 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333550 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 17:47:52.802531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333553 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333556 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333558 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333561 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333564 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333566 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333569 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333572 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333574 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333577 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333579 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333582 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333584 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333587 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333589 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333595 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333597 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333600 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333602 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333605 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 17:47:52.804133 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333608 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333610 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333613 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333615 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333618 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333638 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333642 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333648 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333654 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333658 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333663 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333667 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333671 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333677 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333681 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333686 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333690 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333693 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333697 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 17:47:52.804993 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333701 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333706 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333710 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333714 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333718 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333722 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333726 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333730 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333738 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333743 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333748 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333753 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333757 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333761 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333765 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333769 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333773 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333777 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333783 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 17:47:52.806256 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333787 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333791 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333795 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333801 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333805 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333809 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333813 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:52.333817 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.333824 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.333971 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.336685 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.338021 2554 server.go:1019] "Starting client certificate rotation" Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.338117 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 17:47:52.812775 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.338158 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.340984 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.343845 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.358687 2554 log.go:25] "Validated CRI v1 runtime API" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.360908 2554 log.go:25] "Validated CRI v1 image API" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.361725 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.364171 2554 fs.go:135] Filesystem UUIDs: map[0a3f999c-abb0-48c0-acf8-5674cffba9c6:/dev/nvme0n1p4 55d73607-ec84-4701-a477-393bf2d9876a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 20 17:47:52.813199 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.364186 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.369258 2554 manager.go:216] Machine: {Timestamp:2026-07-20 17:47:52.368033852 +0000 UTC m=+0.306312132 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097708 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d5193f85a154b07575970ba187402 SystemUUID:ec2d5193-f85a-154b-0757-5970ba187402 BootID:fef13364-1c0c-433a-8975-52b0f074581c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:85:87:84:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:85:87:84:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:d8:e8:d0:24:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.369366 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.369446 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.372264 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.372290 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.373533 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.373551 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.373558 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.373569 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.375041 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.381576 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.381704 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382329 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382339 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382352 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382361 2554 kubelet.go:398] "Adding apiserver pod source" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382371 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 17:47:52.815216 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382732 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.382746 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.384728 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.386302 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.386330 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387545 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387558 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387564 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387569 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387578 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387584 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387592 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387597 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387603 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387609 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387621 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.387645 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.388320 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.388326 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.394969 2554 server.go:1268] "Started kubelet" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.395027 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.398710 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.399224 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.399298 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.400934 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.401389 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.403756 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a978082275 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.394875509 +0000 UTC m=+0.333153790,LastTimestamp:2026-07-20 17:47:52.394875509 +0000 UTC m=+0.333153790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.405121 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.408768 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.408795 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.408868 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-108.ec2.internal\" not found" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409463 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409515 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409533 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409714 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409724 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409842 2554 factory.go:55] Registering systemd factory Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.409876 2554 factory.go:223] Registration of the systemd container factory successfully Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410058 2554 factory.go:153] Registering CRI-O factory Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410068 2554 factory.go:223] Registration of the crio container factory successfully Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410112 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410129 2554 factory.go:103] Registering Raw factory Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410142 2554 manager.go:1196] Started watching for new ooms in manager Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.410609 2554 manager.go:349] Starting recovery of all containers Jul 20 17:47:52.816018 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.417895 2554 manager.go:354] Recovery completed Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.423258 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.424060 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.425903 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.426223 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.426236 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.426370 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.427122 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.427153 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.427169 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.428388 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.428398 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.428403 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.428413 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.429406 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.429436 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.429452 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.430570 2554 policy_none.go:49] "None policy: Start" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.430587 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.430601 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.433305 2554 policy_none.go:47] "Start" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.439801 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f47623 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427140643 +0000 UTC m=+0.365418926,LastTimestamp:2026-07-20 17:47:52.427140643 +0000 UTC m=+0.365418926,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.817207 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.451404 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4c131 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427159857 +0000 UTC m=+0.365438139,LastTimestamp:2026-07-20 17:47:52.427159857 +0000 UTC m=+0.365438139,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.461894 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4fa7a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427174522 +0000 UTC m=+0.365452804,LastTimestamp:2026-07-20 17:47:52.427174522 +0000 UTC m=+0.365452804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.477664 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f47623\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f47623 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427140643 +0000 UTC m=+0.365418926,LastTimestamp:2026-07-20 17:47:52.429421829 +0000 UTC m=+0.367700114,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.491606 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f4c131\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4c131 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427159857 +0000 UTC m=+0.365438139,LastTimestamp:2026-07-20 17:47:52.429443843 +0000 UTC m=+0.367722125,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.501295 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f4fa7a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4fa7a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427174522 +0000 UTC m=+0.365452804,LastTimestamp:2026-07-20 17:47:52.42945736 +0000 UTC m=+0.367735642,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504278 2554 manager.go:337] "Starting Device Plugin manager" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.504315 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504327 2554 server.go:87] "Starting device plugin registration server" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504622 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504649 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504751 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504835 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.504843 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.505317 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 17:47:52.820076 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.505362 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-108.ec2.internal\" not found" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.518247 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a97eae660a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.506435082 +0000 UTC m=+0.444713353,LastTimestamp:2026-07-20 17:47:52.506435082 +0000 UTC m=+0.444713353,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.542229 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.543410 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.543445 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.543464 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.543502 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.555591 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.605670 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.606710 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.606741 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.606756 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.606779 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.610904 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fknmh" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.615614 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f47623\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f47623 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427140643 +0000 UTC m=+0.365418926,LastTimestamp:2026-07-20 17:47:52.606723695 +0000 UTC m=+0.545001976,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.619278 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f4c131\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4c131 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427159857 +0000 UTC m=+0.365438139,LastTimestamp:2026-07-20 17:47:52.606747933 +0000 UTC m=+0.545026218,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.820887 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.619668 2554 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.624295 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f4fa7a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f4fa7a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427174522 +0000 UTC m=+0.365452804,LastTimestamp:2026-07-20 17:47:52.606760715 +0000 UTC m=+0.545038995,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.630976 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fknmh" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.631421 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.644160 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal"] Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.644215 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.645115 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.645141 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.645150 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.647298 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.647406 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.647431 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:52.647602 2554 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-108.ec2.internal.18c410a979f47623\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-108.ec2.internal.18c410a979f47623 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-108.ec2.internal,UID:ip-10-0-135-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-108.ec2.internal,},FirstTimestamp:2026-07-20 17:47:52.427140643 +0000 UTC m=+0.365418926,LastTimestamp:2026-07-20 17:47:52.645127552 +0000 UTC m=+0.583405833,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-108.ec2.internal,}" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648005 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648024 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648033 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648043 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648066 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.648078 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.650130 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.650155 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.651374 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.651402 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:52.822915 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:52.651416 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:53.159273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.133087 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:53.159273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.134156 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:53.159273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.134188 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:53.159273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.134201 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:53.159273 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:53.145714 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-108.ec2.internal\" not found" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.408840 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.345227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.408840 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.345294 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.408840 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.366484 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.408840 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:53.366506 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-108.ec2.internal\": node \"ip-10-0-135-108.ec2.internal\" not found" Jul 20 17:47:53.795350 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.794076 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.888287 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.568607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.900942 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.900919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.901076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.900947 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c25f52658a60d1f1fe04d4fdc45521eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-108.ec2.internal\" (UID: \"c25f52658a60d1f1fe04d4fdc45521eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.901076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.900970 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 17:42:52 +0000 UTC" deadline="2028-04-28 19:42:43.331389594 +0000 UTC" Jul 20 17:47:53.901076 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.901024 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15553h54m49.430371297s" Jul 20 17:47:53.901190 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.901071 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 17:47:53.902476 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.901528 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 17:47:53.902476 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:53.901810 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{}\" for node \"ip-10-0-135-108.ec2.internal\": Patch \"https://ade0d5179790e46f791c61b589a33d84-3862e67a95eca7be.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-108.ec2.internal/status?timeout=10s\": http2: client connection force closed via ClientConn.Close" Jul 20 17:47:53.904302 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.904275 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientMemory" Jul 20 17:47:53.904422 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.904307 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 17:47:53.904422 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.904321 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeHasSufficientPID" Jul 20 17:47:53.906420 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:53.906396 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-108.ec2.internal\" not found" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.911881 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:53.911865 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-108.ec2.internal\" not found" node="ip-10-0-135-108.ec2.internal" Jul 20 17:47:53.945046 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.944964 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:47:53.946568 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:53.946547 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:47:54.004171 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.004131 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.004171 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.004172 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c25f52658a60d1f1fe04d4fdc45521eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-108.ec2.internal\" (UID: \"c25f52658a60d1f1fe04d4fdc45521eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.004342 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.004213 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c25f52658a60d1f1fe04d4fdc45521eb-config\") pod \"kube-apiserver-proxy-ip-10-0-135-108.ec2.internal\" (UID: \"c25f52658a60d1f1fe04d4fdc45521eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.004342 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.004244 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea1c278aff3e22a8adafb93d836c5a12-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal\" (UID: \"ea1c278aff3e22a8adafb93d836c5a12\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.009690 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.009674 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.166331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.166293 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:47:54.166726 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.166704 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 17:47:54.167647 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.167615 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.209532 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.209455 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.215920 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.215776 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" Jul 20 17:47:54.270091 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.270061 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 17:47:54.901722 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.901685 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 17:47:54.902129 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.901694 2554 apiserver.go:52] "Watching apiserver" Jul 20 17:47:54.911522 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.911501 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 17:47:54.912566 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.912543 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mtfkc","kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r","openshift-dns/node-resolver-5l5wk","openshift-image-registry/node-ca-78xsn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal","openshift-multus/multus-additional-cni-plugins-9r2vz","openshift-multus/network-metrics-daemon-46fm2","kube-system/konnectivity-agent-t8r5c","openshift-cluster-node-tuning-operator/tuned-bnxq9","openshift-multus/multus-mrspd","openshift-network-diagnostics/network-check-target-dttfw","openshift-network-operator/iptables-alerter-cwhg4"] Jul 20 17:47:54.915007 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.914989 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:54.915105 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:54.915078 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:47:54.917959 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.917942 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:54.920223 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.920199 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:54.923213 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.923198 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:54.926172 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.925664 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:54.927959 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.927373 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.927959 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.927648 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.927959 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.927757 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 17:47:54.929497 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.928431 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfmwj\"" Jul 20 17:47:54.929497 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.928514 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gxl2d\"" Jul 20 17:47:54.929497 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.928603 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 17:47:54.929497 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.928749 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.929755 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.929737 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:54.929876 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:54.929850 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:47:54.933056 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.933035 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:54.935600 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.935579 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:54.936528 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:54.936499 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25f52658a60d1f1fe04d4fdc45521eb.slice/crio-00b5196718f12bfc509794bf5e1296cde0cb64b0219a783dda1d1ec2c514e1f4 WatchSource:0}: Error finding container 00b5196718f12bfc509794bf5e1296cde0cb64b0219a783dda1d1ec2c514e1f4: Status 404 returned error can't find the container with id 00b5196718f12bfc509794bf5e1296cde0cb64b0219a783dda1d1ec2c514e1f4 Jul 20 17:47:54.938688 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.938670 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mrspd" Jul 20 17:47:54.941154 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.941135 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:54.941254 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.941209 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.941254 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:54.941204 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:47:54.942858 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.942814 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qspx\"" Jul 20 17:47:54.942945 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.942873 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.943144 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943121 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 17:47:54.943236 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943210 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 17:47:54.943328 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943305 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 17:47:54.943465 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943441 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.943551 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943445 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.943831 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943696 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.943831 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.943704 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:54.944278 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.944261 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 17:47:54.944560 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.944542 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n82g4\"" Jul 20 17:47:54.944965 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.944948 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.945140 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.945063 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.945600 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.945577 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8trf\"" Jul 20 17:47:54.945861 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.945842 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-k7dh8"] Jul 20 17:47:54.947693 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.947146 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-52jkk\"" Jul 20 17:47:54.947693 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.947354 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 17:47:54.947693 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.947586 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8zzvs\"" Jul 20 17:47:54.947911 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.947876 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 17:47:54.947911 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.947905 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 17:47:54.949427 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.949409 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:54.950460 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:54.950435 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1c278aff3e22a8adafb93d836c5a12.slice/crio-61f62d4ebd86ef24da09190b12cb575dd27bfdbf6c70926c32a570b856d10a82 WatchSource:0}: Error finding container 61f62d4ebd86ef24da09190b12cb575dd27bfdbf6c70926c32a570b856d10a82: Status 404 returned error can't find the container with id 61f62d4ebd86ef24da09190b12cb575dd27bfdbf6c70926c32a570b856d10a82 Jul 20 17:47:54.958886 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.958870 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 17:47:54.959209 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.959194 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.960367 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.960349 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 17:47:54.960485 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.960470 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 17:47:54.960733 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.960719 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 17:47:54.967754 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.967725 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 17:47:54.968168 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.968148 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:47:54.968258 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.968167 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxztv\"" Jul 20 17:47:54.968346 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.968326 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.968408 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.968368 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 17:47:54.968517 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.968452 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7vvt7\"" Jul 20 17:47:54.995045 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:54.995022 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 17:47:55.013096 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.013068 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 17:47:55.013854 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.013830 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 17:47:55.015009 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.014986 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015133 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015022 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015133 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015050 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-env-overrides\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015133 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015112 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-host\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.015256 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-bin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015256 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015207 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-conf-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015256 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-system-cni-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.015373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015301 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-config\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015326 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e470a3-89d3-4902-9441-1b742929dbed-ovn-node-metrics-cert\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015345 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-kubernetes\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.015373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015360 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-cni-binary-copy\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015377 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7dff9a9-ab37-4319-bebc-799e402d6e2c-tmp-dir\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015410 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cnibin\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015441 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-os-release\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015472 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-node-log\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015496 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015531 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-os-release\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015558 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-systemd\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015583 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-tmp\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015604 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpjc\" (UniqueName: \"kubernetes.io/projected/084c96d0-3337-4ff6-8122-73354392e16c-kube-api-access-fjpjc\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015628 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-netns\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015663 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015690 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baa865ba-f651-4070-8e45-026a0bda5a09-serviceca\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015704 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-etc-kubernetes\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015735 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015750 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48b3c481-f7fb-4636-8558-f8d08ec11f38-host-slash\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015785 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-ovn\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9cff\" (UniqueName: \"kubernetes.io/projected/c9e470a3-89d3-4902-9441-1b742929dbed-kube-api-access-g9cff\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.015836 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015838 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-kubelet-config\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015864 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7dff9a9-ab37-4319-bebc-799e402d6e2c-hosts-file\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015912 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baa865ba-f651-4070-8e45-026a0bda5a09-host\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015938 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7v8v\" (UniqueName: \"kubernetes.io/projected/fa6ae863-c428-499f-8d98-a2a8b36edfeb-kube-api-access-z7v8v\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015962 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-var-lib-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.015993 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-lib-modules\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016021 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-var-lib-kubelet\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016038 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-cnibin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-k8s-cni-cncf-io\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016103 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016122 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-bin\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016143 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-netd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fe3b281-c05a-47a1-80c2-696c417b0a48-konnectivity-ca\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016185 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016201 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-socket-dir-parent\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-multus-certs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016264 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-dbus\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016280 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ht2\" (UniqueName: \"kubernetes.io/projected/baa865ba-f651-4070-8e45-026a0bda5a09-kube-api-access-k7ht2\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016294 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-hostroot\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016330 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016368 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9gn\" (UniqueName: \"kubernetes.io/projected/d7dff9a9-ab37-4319-bebc-799e402d6e2c-kube-api-access-xq9gn\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016388 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016404 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016423 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-systemd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016465 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-conf\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016490 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-tuned\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016509 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-kubelet\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpbs\" (UniqueName: \"kubernetes.io/projected/498e6195-611a-45bf-a575-8d80a58307b8-kube-api-access-kvpbs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016553 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.016891 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-multus\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-kubelet\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016619 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fe3b281-c05a-47a1-80c2-696c417b0a48-agent-certs\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016633 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-run\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016666 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-multus-daemon-config\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016690 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016712 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48b3c481-f7fb-4636-8558-f8d08ec11f38-iptables-alerter-script\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016738 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkpc\" (UniqueName: \"kubernetes.io/projected/48b3c481-f7fb-4636-8558-f8d08ec11f38-kube-api-access-bdkpc\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016758 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-systemd-units\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016774 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysconfig\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016790 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-sys\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016805 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll99c\" (UniqueName: \"kubernetes.io/projected/dae1e554-ab06-409b-8ad1-8eec5ec88920-kube-api-access-ll99c\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016820 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-system-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016834 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-device-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016851 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-slash\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-etc-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-script-lib\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016909 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-modprobe-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.017853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016923 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mkf\" (UniqueName: \"kubernetes.io/projected/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kube-api-access-f4mkf\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.017853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016944 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-log-socket\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.017853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.016962 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-netns\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.034235 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.034214 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m94d8" Jul 20 17:47:55.083273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.083202 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m94d8" Jul 20 17:47:55.117826 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117802 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117850 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-env-overrides\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117873 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-host\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117897 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-bin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117914 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.117974 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.117930 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118016 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-conf-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118046 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-system-cni-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118057 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-host\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118065 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-conf-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118074 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-config\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-bin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118101 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e470a3-89d3-4902-9441-1b742929dbed-ovn-node-metrics-cert\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118121 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-system-cni-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118127 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-kubernetes\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118157 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-cni-binary-copy\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7dff9a9-ab37-4319-bebc-799e402d6e2c-tmp-dir\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cnibin\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118321 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-kubernetes\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-os-release\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118392 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-node-log\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118405 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-env-overrides\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118424 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118467 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-os-release\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-systemd\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118512 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-tmp\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118534 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpjc\" (UniqueName: \"kubernetes.io/projected/084c96d0-3337-4ff6-8122-73354392e16c-kube-api-access-fjpjc\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118557 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-netns\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118578 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118601 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baa865ba-f651-4070-8e45-026a0bda5a09-serviceca\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118622 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-etc-kubernetes\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118642 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118646 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7dff9a9-ab37-4319-bebc-799e402d6e2c-tmp-dir\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.118753 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118670 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118695 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-os-release\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118650 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-config\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118718 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48b3c481-f7fb-4636-8558-f8d08ec11f38-host-slash\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118721 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-node-log\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118766 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cnibin\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118801 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-ovn\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118815 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-os-release\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118811 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-cni-binary-copy\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9cff\" (UniqueName: \"kubernetes.io/projected/c9e470a3-89d3-4902-9441-1b742929dbed-kube-api-access-g9cff\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118821 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118858 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-systemd\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-etc-kubernetes\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118860 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-kubelet-config\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-netns\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118910 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-kubelet-config\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118907 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7dff9a9-ab37-4319-bebc-799e402d6e2c-hosts-file\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118936 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48b3c481-f7fb-4636-8558-f8d08ec11f38-host-slash\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.119585 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118965 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7dff9a9-ab37-4319-bebc-799e402d6e2c-hosts-file\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.118986 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119005 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119069 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-ovn\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119068 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.119118 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baa865ba-f651-4070-8e45-026a0bda5a09-host\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119160 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baa865ba-f651-4070-8e45-026a0bda5a09-host\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119180 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7v8v\" (UniqueName: \"kubernetes.io/projected/fa6ae863-c428-499f-8d98-a2a8b36edfeb-kube-api-access-z7v8v\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.119259 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:47:55.619210909 +0000 UTC m=+3.557489209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-var-lib-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119299 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119381 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baa865ba-f651-4070-8e45-026a0bda5a09-serviceca\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119404 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-lib-modules\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119435 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-var-lib-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119477 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-var-lib-kubelet\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-cnibin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.120464 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119523 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-var-lib-kubelet\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119552 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-k8s-cni-cncf-io\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119579 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-cnibin\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119579 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-lib-modules\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119609 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119629 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-k8s-cni-cncf-io\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119691 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-bin\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-netd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119752 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-bin\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119783 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fe3b281-c05a-47a1-80c2-696c417b0a48-konnectivity-ca\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119803 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119851 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-socket-dir-parent\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119853 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-cni-netd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-multus-certs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119919 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-multus-certs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.121331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.119989 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120029 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-dbus\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ht2\" (UniqueName: \"kubernetes.io/projected/baa865ba-f651-4070-8e45-026a0bda5a09-kube-api-access-k7ht2\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120084 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-hostroot\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48680f07-ab02-4e7f-8416-949da11961ba-dbus\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120126 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120155 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120164 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa6ae863-c428-499f-8d98-a2a8b36edfeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120179 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ae863-c428-499f-8d98-a2a8b36edfeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120132 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-hostroot\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120223 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9gn\" (UniqueName: \"kubernetes.io/projected/d7dff9a9-ab37-4319-bebc-799e402d6e2c-kube-api-access-xq9gn\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120250 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120272 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120293 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.120381 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120433 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fe3b281-c05a-47a1-80c2-696c417b0a48-konnectivity-ca\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120438 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-systemd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.122276 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120467 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-conf\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.120487 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:47:55.620474605 +0000 UTC m=+3.558752875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120535 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-tuned\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120556 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-kubelet\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120591 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpbs\" (UniqueName: \"kubernetes.io/projected/498e6195-611a-45bf-a575-8d80a58307b8-kube-api-access-kvpbs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120593 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-run-systemd\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120619 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120555 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-multus-socket-dir-parent\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120645 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-multus\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120688 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-kubelet\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120692 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120717 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-kubelet\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120761 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-var-lib-cni-multus\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120786 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-kubelet\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120811 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fe3b281-c05a-47a1-80c2-696c417b0a48-agent-certs\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-run\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120590 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysctl-conf\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123230 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120867 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-multus-daemon-config\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120894 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120910 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-run\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120920 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48b3c481-f7fb-4636-8558-f8d08ec11f38-iptables-alerter-script\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.120978 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkpc\" (UniqueName: \"kubernetes.io/projected/48b3c481-f7fb-4636-8558-f8d08ec11f38-kube-api-access-bdkpc\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121007 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-systemd-units\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysconfig\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-sys\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121097 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-systemd-units\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121105 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll99c\" (UniqueName: \"kubernetes.io/projected/dae1e554-ab06-409b-8ad1-8eec5ec88920-kube-api-access-ll99c\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121151 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-system-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-device-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121206 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-sysconfig\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121257 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-slash\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121261 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-system-cni-dir\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121212 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-slash\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.123962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121300 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-etc-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121313 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-device-dir\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121327 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121355 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-script-lib\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121358 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-sys\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-modprobe-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mkf\" (UniqueName: \"kubernetes.io/projected/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kube-api-access-f4mkf\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121385 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121441 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-log-socket\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121413 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/498e6195-611a-45bf-a575-8d80a58307b8-multus-daemon-config\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121459 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-modprobe-d\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121356 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-etc-openvswitch\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121508 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e470a3-89d3-4902-9441-1b742929dbed-log-socket\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48b3c481-f7fb-4636-8558-f8d08ec11f38-iptables-alerter-script\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121569 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-netns\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121657 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/498e6195-611a-45bf-a575-8d80a58307b8-host-run-netns\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.121854 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e470a3-89d3-4902-9441-1b742929dbed-ovnkube-script-lib\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124483 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.123006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-tmp\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.124995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.123185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e470a3-89d3-4902-9441-1b742929dbed-ovn-node-metrics-cert\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.124995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.123540 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dae1e554-ab06-409b-8ad1-8eec5ec88920-etc-tuned\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.124995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.123966 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fe3b281-c05a-47a1-80c2-696c417b0a48-agent-certs\") pod \"konnectivity-agent-t8r5c\" (UID: \"4fe3b281-c05a-47a1-80c2-696c417b0a48\") " pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.156443 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.156420 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:47:55.156443 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.156443 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:47:55.156533 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.156454 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:55.156533 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.156506 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:47:55.656492982 +0000 UTC m=+3.594771251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:55.163908 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.163871 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7v8v\" (UniqueName: \"kubernetes.io/projected/fa6ae863-c428-499f-8d98-a2a8b36edfeb-kube-api-access-z7v8v\") pod \"multus-additional-cni-plugins-9r2vz\" (UID: \"fa6ae863-c428-499f-8d98-a2a8b36edfeb\") " pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.163908 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.163891 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkpc\" (UniqueName: \"kubernetes.io/projected/48b3c481-f7fb-4636-8558-f8d08ec11f38-kube-api-access-bdkpc\") pod \"iptables-alerter-cwhg4\" (UID: \"48b3c481-f7fb-4636-8558-f8d08ec11f38\") " pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.164354 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.164336 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpjc\" (UniqueName: \"kubernetes.io/projected/084c96d0-3337-4ff6-8122-73354392e16c-kube-api-access-fjpjc\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.165809 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.165783 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll99c\" (UniqueName: \"kubernetes.io/projected/dae1e554-ab06-409b-8ad1-8eec5ec88920-kube-api-access-ll99c\") pod \"tuned-bnxq9\" (UID: \"dae1e554-ab06-409b-8ad1-8eec5ec88920\") " pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.166318 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.166303 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9cff\" (UniqueName: \"kubernetes.io/projected/c9e470a3-89d3-4902-9441-1b742929dbed-kube-api-access-g9cff\") pod \"ovnkube-node-k7dh8\" (UID: \"c9e470a3-89d3-4902-9441-1b742929dbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.166369 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.166315 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9gn\" (UniqueName: \"kubernetes.io/projected/d7dff9a9-ab37-4319-bebc-799e402d6e2c-kube-api-access-xq9gn\") pod \"node-resolver-5l5wk\" (UID: \"d7dff9a9-ab37-4319-bebc-799e402d6e2c\") " pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.166689 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.166667 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpbs\" (UniqueName: \"kubernetes.io/projected/498e6195-611a-45bf-a575-8d80a58307b8-kube-api-access-kvpbs\") pod \"multus-mrspd\" (UID: \"498e6195-611a-45bf-a575-8d80a58307b8\") " pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.169601 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.169582 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ht2\" (UniqueName: \"kubernetes.io/projected/baa865ba-f651-4070-8e45-026a0bda5a09-kube-api-access-k7ht2\") pod \"node-ca-78xsn\" (UID: \"baa865ba-f651-4070-8e45-026a0bda5a09\") " pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.171378 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.171361 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mkf\" (UniqueName: \"kubernetes.io/projected/ea095938-4bcc-4b94-a273-0ebe43d2e2f1-kube-api-access-f4mkf\") pod \"aws-ebs-csi-driver-node-gp42r\" (UID: \"ea095938-4bcc-4b94-a273-0ebe43d2e2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.242817 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.242777 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" Jul 20 17:47:55.251693 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.251671 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5l5wk" Jul 20 17:47:55.253343 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.253319 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea095938_4bcc_4b94_a273_0ebe43d2e2f1.slice/crio-a87037b8ea3aa31d9e238ce07b9a01104e4540b460e4e16a44942fe6d567eb88 WatchSource:0}: Error finding container a87037b8ea3aa31d9e238ce07b9a01104e4540b460e4e16a44942fe6d567eb88: Status 404 returned error can't find the container with id a87037b8ea3aa31d9e238ce07b9a01104e4540b460e4e16a44942fe6d567eb88 Jul 20 17:47:55.258716 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.258691 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-78xsn" Jul 20 17:47:55.260685 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.260660 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7dff9a9_ab37_4319_bebc_799e402d6e2c.slice/crio-1f7f0c4a800563a33c982704c27ec9f0e0524e9ba3a10954341e364e3c290d17 WatchSource:0}: Error finding container 1f7f0c4a800563a33c982704c27ec9f0e0524e9ba3a10954341e364e3c290d17: Status 404 returned error can't find the container with id 1f7f0c4a800563a33c982704c27ec9f0e0524e9ba3a10954341e364e3c290d17 Jul 20 17:47:55.264243 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.264228 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" Jul 20 17:47:55.267982 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.267959 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa865ba_f651_4070_8e45_026a0bda5a09.slice/crio-3de5a261cafa8b605eebc1c4cebb5c292fb49535fb2c05fcd1b233fd24d8cc3f WatchSource:0}: Error finding container 3de5a261cafa8b605eebc1c4cebb5c292fb49535fb2c05fcd1b233fd24d8cc3f: Status 404 returned error can't find the container with id 3de5a261cafa8b605eebc1c4cebb5c292fb49535fb2c05fcd1b233fd24d8cc3f Jul 20 17:47:55.270289 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.270258 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:47:55.273547 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.273527 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ae863_c428_499f_8d98_a2a8b36edfeb.slice/crio-3d1d1f1cbc3da9d0218b6128efbe9c1272a1632cc99f77293eba416aa31c0dab WatchSource:0}: Error finding container 3d1d1f1cbc3da9d0218b6128efbe9c1272a1632cc99f77293eba416aa31c0dab: Status 404 returned error can't find the container with id 3d1d1f1cbc3da9d0218b6128efbe9c1272a1632cc99f77293eba416aa31c0dab Jul 20 17:47:55.275263 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.275243 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" Jul 20 17:47:55.279743 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.279725 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe3b281_c05a_47a1_80c2_696c417b0a48.slice/crio-2a00af6eee90e8b4a3e83bb70fb9027ba08ecc0e87c06e8e70ad72cd29042239 WatchSource:0}: Error finding container 2a00af6eee90e8b4a3e83bb70fb9027ba08ecc0e87c06e8e70ad72cd29042239: Status 404 returned error can't find the container with id 2a00af6eee90e8b4a3e83bb70fb9027ba08ecc0e87c06e8e70ad72cd29042239 Jul 20 17:47:55.280899 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.280836 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mrspd" Jul 20 17:47:55.284280 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.284261 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae1e554_ab06_409b_8ad1_8eec5ec88920.slice/crio-4efa41f3514f13e279b6b5087df6553ad0edfbe82be7df36fe68d4e38b319b5d WatchSource:0}: Error finding container 4efa41f3514f13e279b6b5087df6553ad0edfbe82be7df36fe68d4e38b319b5d: Status 404 returned error can't find the container with id 4efa41f3514f13e279b6b5087df6553ad0edfbe82be7df36fe68d4e38b319b5d Jul 20 17:47:55.287195 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.287125 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cwhg4" Jul 20 17:47:55.290461 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.290434 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498e6195_611a_45bf_a575_8d80a58307b8.slice/crio-70777b71d035156a962a2f5686b9f85f2e55866e15184fd3148c42e4138f4f2d WatchSource:0}: Error finding container 70777b71d035156a962a2f5686b9f85f2e55866e15184fd3148c42e4138f4f2d: Status 404 returned error can't find the container with id 70777b71d035156a962a2f5686b9f85f2e55866e15184fd3148c42e4138f4f2d Jul 20 17:47:55.291815 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.291792 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:47:55.299123 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.299087 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b3c481_f7fb_4636_8558_f8d08ec11f38.slice/crio-256daea1384dbce745ab259775cd3e361cdf41f2839283423aefa97af37a914a WatchSource:0}: Error finding container 256daea1384dbce745ab259775cd3e361cdf41f2839283423aefa97af37a914a: Status 404 returned error can't find the container with id 256daea1384dbce745ab259775cd3e361cdf41f2839283423aefa97af37a914a Jul 20 17:47:55.301961 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:47:55.301940 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e470a3_89d3_4902_9441_1b742929dbed.slice/crio-a8b4bfbf6e6b54cc3227d763487a59bb470b9be2b24144e6fd54caac64319133 WatchSource:0}: Error finding container a8b4bfbf6e6b54cc3227d763487a59bb470b9be2b24144e6fd54caac64319133: Status 404 returned error can't find the container with id a8b4bfbf6e6b54cc3227d763487a59bb470b9be2b24144e6fd54caac64319133 Jul 20 17:47:55.624945 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.624907 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:55.625159 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.624975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:55.625159 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.625084 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:55.625159 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.625086 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:55.625159 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.625143 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:47:56.625128822 +0000 UTC m=+4.563407092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:55.625159 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.625158 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:47:56.625151792 +0000 UTC m=+4.563430061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:55.726202 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.726160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:55.726379 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.726334 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:47:55.726379 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.726354 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:47:55.726379 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.726366 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:55.726559 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:55.726428 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:47:56.726409346 +0000 UTC m=+4.664687631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:55.909338 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.909066 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"a8b4bfbf6e6b54cc3227d763487a59bb470b9be2b24144e6fd54caac64319133"} Jul 20 17:47:55.913988 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.913906 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mrspd" event={"ID":"498e6195-611a-45bf-a575-8d80a58307b8","Type":"ContainerStarted","Data":"70777b71d035156a962a2f5686b9f85f2e55866e15184fd3148c42e4138f4f2d"} Jul 20 17:47:55.915534 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.915452 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" event={"ID":"dae1e554-ab06-409b-8ad1-8eec5ec88920","Type":"ContainerStarted","Data":"4efa41f3514f13e279b6b5087df6553ad0edfbe82be7df36fe68d4e38b319b5d"} Jul 20 17:47:55.919381 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.919292 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t8r5c" event={"ID":"4fe3b281-c05a-47a1-80c2-696c417b0a48","Type":"ContainerStarted","Data":"2a00af6eee90e8b4a3e83bb70fb9027ba08ecc0e87c06e8e70ad72cd29042239"} Jul 20 17:47:55.924542 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.924479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerStarted","Data":"3d1d1f1cbc3da9d0218b6128efbe9c1272a1632cc99f77293eba416aa31c0dab"} Jul 20 17:47:55.941191 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.941111 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-78xsn" event={"ID":"baa865ba-f651-4070-8e45-026a0bda5a09","Type":"ContainerStarted","Data":"3de5a261cafa8b605eebc1c4cebb5c292fb49535fb2c05fcd1b233fd24d8cc3f"} Jul 20 17:47:55.945737 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.945648 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5l5wk" event={"ID":"d7dff9a9-ab37-4319-bebc-799e402d6e2c","Type":"ContainerStarted","Data":"1f7f0c4a800563a33c982704c27ec9f0e0524e9ba3a10954341e364e3c290d17"} Jul 20 17:47:55.951227 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.951151 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" event={"ID":"ea095938-4bcc-4b94-a273-0ebe43d2e2f1","Type":"ContainerStarted","Data":"a87037b8ea3aa31d9e238ce07b9a01104e4540b460e4e16a44942fe6d567eb88"} Jul 20 17:47:55.958360 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.958314 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cwhg4" event={"ID":"48b3c481-f7fb-4636-8558-f8d08ec11f38","Type":"ContainerStarted","Data":"256daea1384dbce745ab259775cd3e361cdf41f2839283423aefa97af37a914a"} Jul 20 17:47:55.966460 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.965980 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" event={"ID":"ea1c278aff3e22a8adafb93d836c5a12","Type":"ContainerStarted","Data":"61f62d4ebd86ef24da09190b12cb575dd27bfdbf6c70926c32a570b856d10a82"} Jul 20 17:47:55.969591 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:55.969566 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" event={"ID":"c25f52658a60d1f1fe04d4fdc45521eb","Type":"ContainerStarted","Data":"00b5196718f12bfc509794bf5e1296cde0cb64b0219a783dda1d1ec2c514e1f4"} Jul 20 17:47:56.088025 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.087972 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:42:55 +0000 UTC" deadline="2028-01-11 10:17:05.790461935 +0000 UTC" Jul 20 17:47:56.088025 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.088016 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12952h29m9.702451387s" Jul 20 17:47:56.545438 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.544774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:56.545438 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.544929 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:47:56.545438 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.544774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:56.545438 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.545394 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:47:56.546859 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.546838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:56.547006 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.546928 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.633669 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.633739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.633978 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.634046 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:47:58.634026555 +0000 UTC m=+6.572304848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.634475 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:56.634564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.634524 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:47:58.634509072 +0000 UTC m=+6.572787348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:56.735047 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:56.734376 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:56.735047 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.734562 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:47:56.735047 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.734583 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:47:56.735047 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.734596 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:56.735047 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:56.734662 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:47:58.734640037 +0000 UTC m=+6.672918334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:57.089032 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:57.088549 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:42:55 +0000 UTC" deadline="2028-01-12 14:21:45.208228722 +0000 UTC" Jul 20 17:47:57.089032 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:57.088588 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12980h33m48.119645365s" Jul 20 17:47:58.547150 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.547120 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:58.547792 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.547286 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:47:58.548424 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.548040 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:58.548424 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.548149 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:47:58.548424 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.548208 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:58.548424 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.548276 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.653516 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.653607 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.653765 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.653830 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:48:02.653810225 +0000 UTC m=+10.592088515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.654216 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:58.654305 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.654266 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:48:02.654250834 +0000 UTC m=+10.592529117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:47:58.755348 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:47:58.754749 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:47:58.755348 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.754922 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:47:58.755348 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.754940 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:47:58.755348 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.754952 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:47:58.755348 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:47:58.755009 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:02.754990304 +0000 UTC m=+10.693268574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:00.544747 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:00.544885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:00.544900 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:00.545016 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:00.545060 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:00.545134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:00.545126 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:02.545398 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.545318 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:02.545880 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.545457 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:02.545880 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.545516 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:02.545880 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.545706 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:02.545880 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.545774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:02.545880 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.545866 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.686177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.686246 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.686390 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.686453 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:48:10.686432785 +0000 UTC m=+18.624711077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.686852 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:02.686942 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.686907 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:48:10.686889988 +0000 UTC m=+18.625168279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:02.786948 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:02.786909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:02.787109 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.787073 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:02.787109 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.787093 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:02.787109 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.787106 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:02.787283 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:02.787180 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:10.787153484 +0000 UTC m=+18.725431751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:04.544517 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:04.544476 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:04.544969 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:04.544476 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:04.544969 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:04.544639 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:04.544969 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:04.544737 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:04.544969 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:04.544767 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:04.544969 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:04.544814 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:06.544298 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:06.544109 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:06.544298 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:06.544109 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:06.544298 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:06.544233 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:06.544298 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:06.544262 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:06.544860 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:06.544352 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:06.544860 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:06.544440 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:08.544120 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:08.544081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:08.544120 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:08.544111 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:08.544718 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:08.544081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:08.544718 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:08.544227 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:08.544718 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:08.544316 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:08.544718 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:08.544395 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:10.544406 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.544363 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:10.544406 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.544404 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:10.544988 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.544371 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:10.544988 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.544527 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:10.544988 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.544605 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:10.544988 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.544708 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:10.747926 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.747892 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:10.748126 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.747937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:10.748126 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.748074 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:10.748126 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.748117 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:10.748259 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.748157 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:48:26.748136008 +0000 UTC m=+34.686414292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:10.748259 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.748178 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:48:26.748167372 +0000 UTC m=+34.686445640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:10.849209 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:10.849121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:10.849381 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.849293 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:10.849381 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.849315 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:10.849381 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.849328 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:10.849540 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:10.849390 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:26.849370495 +0000 UTC m=+34.787648767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:12.544968 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:12.544935 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:12.545438 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:12.545045 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:12.545438 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:12.545054 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:12.545438 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:12.545068 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:12.545438 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:12.545159 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:12.545438 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:12.545242 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:14.544732 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:14.544682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:14.545190 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:14.544809 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:14.545190 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:14.544682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:14.545347 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:14.545263 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:14.545347 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:14.545286 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:14.545424 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:14.545374 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:16.021851 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.021597 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" event={"ID":"c25f52658a60d1f1fe04d4fdc45521eb","Type":"ContainerStarted","Data":"fc1d65a955c3e24d8fb9310e0d977ea423c9b2281ee4dbd7756b0dd79eb9a8f3"} Jul 20 17:48:16.023315 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.023297 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:48:16.023596 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.023576 2554 generic.go:356] "Generic (PLEG): container finished" podID="c9e470a3-89d3-4902-9441-1b742929dbed" containerID="81c6e06612655a400b04fd2aed25b34203b21e47093b4dea0127b855d2c316f8" exitCode=1 Jul 20 17:48:16.023687 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.023665 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"9e2e6d1765265a93287a7d7bb216e01ddcb77767bf1e5ee44651d67454deb121"} Jul 20 17:48:16.023764 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.023750 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerDied","Data":"81c6e06612655a400b04fd2aed25b34203b21e47093b4dea0127b855d2c316f8"} Jul 20 17:48:16.023819 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.023786 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"d29d2485185af323b27ddf88646262af86006c67b978ef690232563f37064d78"} Jul 20 17:48:16.025931 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.025908 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mrspd" event={"ID":"498e6195-611a-45bf-a575-8d80a58307b8","Type":"ContainerStarted","Data":"e910a13e0f26e24f4ee67189b29d62204d3fe76f6a07d80539c0a524dec86939"} Jul 20 17:48:16.027274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.027256 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" event={"ID":"dae1e554-ab06-409b-8ad1-8eec5ec88920","Type":"ContainerStarted","Data":"4f0f6692ed424d79d2927ab1ecce59fd5222cf6b6232bca4cf5e87f08d9da02c"} Jul 20 17:48:16.048823 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.048768 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-108.ec2.internal" podStartSLOduration=22.048748688 podStartE2EDuration="22.048748688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:16.048180816 +0000 UTC m=+23.986459126" watchObservedRunningTime="2026-07-20 17:48:16.048748688 +0000 UTC m=+23.987026981" Jul 20 17:48:16.121121 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.119980 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bnxq9" podStartSLOduration=1.962569155 podStartE2EDuration="22.11994371s" totalImagesPullingTime="20.157374555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:16.082156045 +0000 UTC m=+24.020434366" watchObservedRunningTime="2026-07-20 17:48:16.11994371 +0000 UTC m=+24.058222003" Jul 20 17:48:16.121121 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.120609 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mrspd" podStartSLOduration=1.9338906040000001 podStartE2EDuration="22.12059581s" totalImagesPullingTime="20.186705206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:16.11970401 +0000 UTC m=+24.057982325" watchObservedRunningTime="2026-07-20 17:48:16.12059581 +0000 UTC m=+24.058874101" Jul 20 17:48:16.544461 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.544418 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:16.544658 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.544436 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:16.544658 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:16.544563 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:16.544658 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:16.544611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:16.544818 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:16.544747 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:16.544818 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:16.544810 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:17.030909 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.030822 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cwhg4" event={"ID":"48b3c481-f7fb-4636-8558-f8d08ec11f38","Type":"ContainerStarted","Data":"7f0df1985aab9b6e4f668cb76e2695e096723496ae321db165745ad5b74603c0"} Jul 20 17:48:17.032417 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.032392 2554 generic.go:356] "Generic (PLEG): container finished" podID="ea1c278aff3e22a8adafb93d836c5a12" containerID="3428085bf4ffae8cac75975ffd6d770076a306f63c8be9131f1651fbfb5ec3e0" exitCode=0 Jul 20 17:48:17.032528 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.032479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" event={"ID":"ea1c278aff3e22a8adafb93d836c5a12","Type":"ContainerDied","Data":"3428085bf4ffae8cac75975ffd6d770076a306f63c8be9131f1651fbfb5ec3e0"} Jul 20 17:48:17.035439 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.035421 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:48:17.035803 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.035786 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"15256198d638195670cbcc49e6d6f5fe9494f13f07876102707ad9ed33630710"} Jul 20 17:48:17.035871 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.035855 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"6d2017126720c7844b58622d4fd305824d562414482a227178e60a6fc110bbd2"} Jul 20 17:48:17.035936 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.035923 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"2ff8c8d30c593c25ac0cc83d52b516df96ccf244ab215e3920420e1e5f113791"} Jul 20 17:48:17.037115 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.037095 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t8r5c" event={"ID":"4fe3b281-c05a-47a1-80c2-696c417b0a48","Type":"ContainerStarted","Data":"744ee3ce07790f9ae950b6b3b99aa91e13b35bf34189ef6e4c560b0e299ded5f"} Jul 20 17:48:17.038470 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.038451 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="e2913c49c8193cb959718eea2c967c9c47a558adcf7bb464d55a77f4424e25f5" exitCode=0 Jul 20 17:48:17.038558 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.038515 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"e2913c49c8193cb959718eea2c967c9c47a558adcf7bb464d55a77f4424e25f5"} Jul 20 17:48:17.039829 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.039810 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-78xsn" event={"ID":"baa865ba-f651-4070-8e45-026a0bda5a09","Type":"ContainerStarted","Data":"024a8784db752337e707b16bd8831443e20b569ddfd93e2ffed496088ecff5de"} Jul 20 17:48:17.041003 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.040985 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5l5wk" event={"ID":"d7dff9a9-ab37-4319-bebc-799e402d6e2c","Type":"ContainerStarted","Data":"9b124b4cc3fb4c2b985d8680469f8cd3eb4a65ac75cde8a7a306e0e831ee1f24"} Jul 20 17:48:17.042221 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.042197 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" event={"ID":"ea095938-4bcc-4b94-a273-0ebe43d2e2f1","Type":"ContainerStarted","Data":"6c469fafc33f21a6466117598b27c67b82816a6e52fa2af83f9b17970dffc44c"} Jul 20 17:48:17.120912 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.120737 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cwhg4" podStartSLOduration=2.977631733 podStartE2EDuration="23.120720742s" totalImagesPullingTime="20.143089009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:17.07835077 +0000 UTC m=+25.016629060" watchObservedRunningTime="2026-07-20 17:48:17.120720742 +0000 UTC m=+25.058999032" Jul 20 17:48:17.158120 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.158075 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5l5wk" podStartSLOduration=2.9770787260000002 podStartE2EDuration="23.158059414s" totalImagesPullingTime="20.180980688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:17.157920166 +0000 UTC m=+25.096198455" watchObservedRunningTime="2026-07-20 17:48:17.158059414 +0000 UTC m=+25.096337705" Jul 20 17:48:17.295893 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.295788 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-78xsn" podStartSLOduration=3.122141637 podStartE2EDuration="23.295765519s" totalImagesPullingTime="20.173623882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:17.223788802 +0000 UTC m=+25.162067116" watchObservedRunningTime="2026-07-20 17:48:17.295765519 +0000 UTC m=+25.234043812" Jul 20 17:48:17.904083 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.904059 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 17:48:17.914337 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.914250 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T17:48:17.904076833Z","UUID":"c26132ec-c6e6-4a17-bf9b-6c7e3429421c","Handler":null,"Name":"","Endpoint":""} Jul 20 17:48:17.917035 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.917012 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 17:48:17.917135 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:17.917059 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 17:48:18.046334 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.046251 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" event={"ID":"ea095938-4bcc-4b94-a273-0ebe43d2e2f1","Type":"ContainerStarted","Data":"e41c297c730a136444a5d03796aa92aba4fce4d68a18964b2edf5d4b8fd6f44d"} Jul 20 17:48:18.048225 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.048196 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" event={"ID":"ea1c278aff3e22a8adafb93d836c5a12","Type":"ContainerStarted","Data":"8b26df3191960be13cc7c32f2171950ebf81ae723136f2688e38b9c2d9d92803"} Jul 20 17:48:18.078749 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.078688 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t8r5c" podStartSLOduration=3.949016365 podStartE2EDuration="24.078665267s" totalImagesPullingTime="20.129648902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:17.358916952 +0000 UTC m=+25.297195243" watchObservedRunningTime="2026-07-20 17:48:18.078665267 +0000 UTC m=+26.016943558" Jul 20 17:48:18.544300 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.544260 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:18.544300 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.544301 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:18.544573 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.544260 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:18.544573 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:18.544390 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:18.544573 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:18.544472 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:18.544573 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:18.544542 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:18.687853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.687768 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:48:18.688983 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.688954 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:48:18.713008 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:18.712953 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-108.ec2.internal" podStartSLOduration=24.712938357 podStartE2EDuration="24.712938357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:18.078856262 +0000 UTC m=+26.017134537" watchObservedRunningTime="2026-07-20 17:48:18.712938357 +0000 UTC m=+26.651216647" Jul 20 17:48:19.054965 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:19.054743 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:48:19.055555 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:19.055489 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"8a4ebf33a438336e21033f4938b2da4c4652e7ecaf71afa643e5d29a24f6cfbb"} Jul 20 17:48:19.055958 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:19.055933 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:48:19.056588 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:19.056565 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t8r5c" Jul 20 17:48:20.059936 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:20.059892 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" event={"ID":"ea095938-4bcc-4b94-a273-0ebe43d2e2f1","Type":"ContainerStarted","Data":"7e15f7163d365850bdc1d869bb1121912fa48f810810655052949023dfbe8055"} Jul 20 17:48:20.543943 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:20.543859 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:20.544085 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:20.543870 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:20.544085 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:20.543962 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:20.544085 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:20.543988 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:20.544085 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:20.544079 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:20.544226 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:20.544140 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:21.066349 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:21.066165 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:48:21.066947 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:21.066809 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"7b5a0a71a4b8d18cdfc146997a442a85fd7490a4c7820aa91e7dea01c6484ac2"} Jul 20 17:48:21.067519 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:21.067500 2554 scope.go:117] "RemoveContainer" containerID="81c6e06612655a400b04fd2aed25b34203b21e47093b4dea0127b855d2c316f8" Jul 20 17:48:21.126821 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:21.126778 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gp42r" podStartSLOduration=12.287987613 podStartE2EDuration="27.126761961s" totalImagesPullingTime="14.838774348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:20.095213175 +0000 UTC m=+28.033491464" watchObservedRunningTime="2026-07-20 17:48:21.126761961 +0000 UTC m=+29.065040250" Jul 20 17:48:22.073351 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.073320 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:48:22.073975 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.073698 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" event={"ID":"c9e470a3-89d3-4902-9441-1b742929dbed","Type":"ContainerStarted","Data":"98ad299bd016e8b23bdd8547f77354ac5b2f05ecf3e67f644f2a9634e04c9c62"} Jul 20 17:48:22.074072 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.074053 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:22.074143 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.074133 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:22.074205 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.074186 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:22.092848 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.092820 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:22.093306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.093283 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:22.140968 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.140906 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" podStartSLOduration=6.76416255 podStartE2EDuration="27.140885974s" totalImagesPullingTime="20.376723424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:22.136799829 +0000 UTC m=+30.075078140" watchObservedRunningTime="2026-07-20 17:48:22.140885974 +0000 UTC m=+30.079164268" Jul 20 17:48:22.544608 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.544580 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:22.544787 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.544684 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:22.544787 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:22.544730 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:22.544787 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:22.544756 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:22.544931 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:22.544820 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:22.544931 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:22.544886 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:23.077934 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.077900 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerStarted","Data":"0251834acefb15f6649d702b32acccab59652ac4bd413034b66ab6834a448128"} Jul 20 17:48:23.395277 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.395094 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46fm2"] Jul 20 17:48:23.395447 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.395426 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:23.395550 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:23.395529 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:23.402341 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.402310 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtfkc"] Jul 20 17:48:23.402472 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.402417 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:23.402548 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:23.402502 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:23.402820 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.402800 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dttfw"] Jul 20 17:48:23.402893 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:23.402886 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:23.402972 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:23.402955 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:24.081305 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:24.081270 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="0251834acefb15f6649d702b32acccab59652ac4bd413034b66ab6834a448128" exitCode=0 Jul 20 17:48:24.081767 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:24.081358 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"0251834acefb15f6649d702b32acccab59652ac4bd413034b66ab6834a448128"} Jul 20 17:48:25.544007 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:25.543974 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:25.544461 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:25.544006 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:25.544461 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:25.544071 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:25.544461 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:25.544082 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:25.544461 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:25.544138 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:25.544461 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:25.544215 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:26.088042 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:26.088007 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="1fcfae1e750f77e3a86657df4a938c8583e355a87b2cb46c68e8c811e6929fca" exitCode=0 Jul 20 17:48:26.088200 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:26.088059 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"1fcfae1e750f77e3a86657df4a938c8583e355a87b2cb46c68e8c811e6929fca"} Jul 20 17:48:26.775731 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:26.775701 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:26.776134 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:26.775742 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:26.776134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.775837 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:26.776134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.775838 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:26.776134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.775887 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:48:58.775873789 +0000 UTC m=+66.714152060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 17:48:26.776134 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.775900 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret podName:48680f07-ab02-4e7f-8416-949da11961ba nodeName:}" failed. No retries permitted until 2026-07-20 17:48:58.775893763 +0000 UTC m=+66.714172031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret") pod "global-pull-secret-syncer-mtfkc" (UID: "48680f07-ab02-4e7f-8416-949da11961ba") : object "kube-system"/"original-pull-secret" not registered Jul 20 17:48:26.876534 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:26.876496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:26.876736 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.876715 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 17:48:26.876781 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.876748 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 17:48:26.876781 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.876762 2554 projected.go:196] Error preparing data for projected volume kube-api-access-hlw58 for pod openshift-network-diagnostics/network-check-target-dttfw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:26.876843 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:26.876832 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58 podName:034807dc-20ae-4b6a-8ed6-277f795efba1 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:58.87680785 +0000 UTC m=+66.815086141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hlw58" (UniqueName: "kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58") pod "network-check-target-dttfw" (UID: "034807dc-20ae-4b6a-8ed6-277f795efba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 17:48:27.092745 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:27.092714 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerStarted","Data":"4702f425254a053727665677561f90cbb0740f6c4e0ed9ef11b80105c9347968"} Jul 20 17:48:27.544406 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:27.544363 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:27.544406 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:27.544397 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:27.544619 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:27.544402 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:27.544619 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:27.544490 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dttfw" podUID="034807dc-20ae-4b6a-8ed6-277f795efba1" Jul 20 17:48:27.544619 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:27.544558 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mtfkc" podUID="48680f07-ab02-4e7f-8416-949da11961ba" Jul 20 17:48:27.544736 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:27.544618 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46fm2" podUID="084c96d0-3337-4ff6-8122-73354392e16c" Jul 20 17:48:28.099841 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:28.099805 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="4702f425254a053727665677561f90cbb0740f6c4e0ed9ef11b80105c9347968" exitCode=0 Jul 20 17:48:28.100203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:28.099865 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"4702f425254a053727665677561f90cbb0740f6c4e0ed9ef11b80105c9347968"} Jul 20 17:48:29.310712 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.310614 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-108.ec2.internal" event="NodeReady" Jul 20 17:48:29.311118 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.310756 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 17:48:29.369253 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.369214 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8"] Jul 20 17:48:29.372344 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.372314 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.380587 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.380555 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757ff4758d-ql765"] Jul 20 17:48:29.381914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.381774 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 17:48:29.381914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.381794 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 17:48:29.381914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.381842 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 17:48:29.381914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.381878 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 17:48:29.382179 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.382014 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 17:48:29.382179 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.382117 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 17:48:29.382487 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.382468 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 17:48:29.383806 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.383780 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82"] Jul 20 17:48:29.383921 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.383902 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.387698 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.387672 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6"] Jul 20 17:48:29.387806 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.387766 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.390880 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.390850 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 17:48:29.390880 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.390873 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 17:48:29.391104 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.391082 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.391358 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.391335 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 20 17:48:29.391428 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.391406 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 17:48:29.391598 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.391585 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4fzg\"" Jul 20 17:48:29.396596 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.396575 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 17:48:29.404036 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.404012 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 20 17:48:29.405735 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.405707 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82"] Jul 20 17:48:29.414414 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.414387 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cl6zd\"" Jul 20 17:48:29.417588 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.417562 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t54gs"] Jul 20 17:48:29.421833 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.421810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757ff4758d-ql765"] Jul 20 17:48:29.421955 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.421847 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6"] Jul 20 17:48:29.421955 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.421936 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.425855 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.425834 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 17:48:29.426067 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.426049 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkvwc\"" Jul 20 17:48:29.426379 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.426361 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 17:48:29.435587 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.435554 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8"] Jul 20 17:48:29.448685 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.448653 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t54gs"] Jul 20 17:48:29.465482 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.465446 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nn9qb"] Jul 20 17:48:29.468234 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.468218 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.471741 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.471715 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 17:48:29.471741 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.471734 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v69jf\"" Jul 20 17:48:29.471942 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.471715 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 17:48:29.472061 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.472045 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 17:48:29.484687 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.484663 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn9qb"] Jul 20 17:48:29.493586 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493553 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.493586 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493589 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr69\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493673 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1e5356-333b-49bd-a169-6b2c5f603205-config-volume\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493715 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493758 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.493779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493775 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1e5356-333b-49bd-a169-6b2c5f603205-tmp-dir\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493792 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493831 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493856 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9k4\" (UniqueName: \"kubernetes.io/projected/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-kube-api-access-4c9k4\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493896 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dp8\" (UniqueName: \"kubernetes.io/projected/af74fb5b-46b2-4082-a297-2707fa8f3fbe-kube-api-access-d7dp8\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493920 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9j8\" (UniqueName: \"kubernetes.io/projected/ff1e5356-333b-49bd-a169-6b2c5f603205-kube-api-access-dx9j8\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493949 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493970 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.494004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.493987 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-tmp\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494010 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494032 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494057 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af74fb5b-46b2-4082-a297-2707fa8f3fbe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494081 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhjn\" (UniqueName: \"kubernetes.io/projected/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-kube-api-access-wkhjn\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494110 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.494274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.494125 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.544236 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.544196 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:29.544594 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.544267 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:29.544594 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.544283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:29.552833 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.552341 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 17:48:29.552833 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.552486 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcx42\"" Jul 20 17:48:29.552833 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.552340 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 17:48:29.553211 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.553190 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 17:48:29.553345 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.553323 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjfj5\"" Jul 20 17:48:29.554442 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.554423 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 17:48:29.594771 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594736 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1e5356-333b-49bd-a169-6b2c5f603205-tmp-dir\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.594771 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594771 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594789 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9k4\" (UniqueName: \"kubernetes.io/projected/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-kube-api-access-4c9k4\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594823 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594840 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dp8\" (UniqueName: \"kubernetes.io/projected/af74fb5b-46b2-4082-a297-2707fa8f3fbe-kube-api-access-d7dp8\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594856 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9j8\" (UniqueName: \"kubernetes.io/projected/ff1e5356-333b-49bd-a169-6b2c5f603205-kube-api-access-dx9j8\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594897 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594926 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594947 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-tmp\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594971 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.594996 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.594995 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595015 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af74fb5b-46b2-4082-a297-2707fa8f3fbe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595037 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhjn\" (UniqueName: \"kubernetes.io/projected/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-kube-api-access-wkhjn\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffd3795-b55e-4aa6-a848-934f0f73b203-cert\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1e5356-333b-49bd-a169-6b2c5f603205-tmp-dir\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595307 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595338 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595422 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.595491 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595455 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dth6l\" (UniqueName: \"kubernetes.io/projected/fffd3795-b55e-4aa6-a848-934f0f73b203-kube-api-access-dth6l\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595492 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595739 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595761 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr69\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1e5356-333b-49bd-a169-6b2c5f603205-config-volume\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595821 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595855 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.595903 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.596223 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:29.596017 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:29.596223 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:29.596032 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:29.596223 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:29.596094 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:30.096077161 +0000 UTC m=+38.034355442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:29.596223 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.596165 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.596449 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.595855 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.596449 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:29.596249 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:29.596449 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:29.596336 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:30.096319521 +0000 UTC m=+38.034597804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:29.596690 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.596604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1e5356-333b-49bd-a169-6b2c5f603205-config-volume\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.597209 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.597185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-tmp\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.597993 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.597957 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.600588 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.600588 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600572 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.600588 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600583 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-hub\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.600771 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.600771 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600568 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-ca\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.600771 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600646 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af74fb5b-46b2-4082-a297-2707fa8f3fbe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.601011 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.600994 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-klusterlet-config\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.601193 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.601172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.618963 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.618936 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr69\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.629446 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.629418 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:29.631192 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.631163 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9k4\" (UniqueName: \"kubernetes.io/projected/5e1b9d20-2acf-4f1c-a45a-a4947dc8853b-kube-api-access-4c9k4\") pod \"klusterlet-addon-workmgr-68cdcc9569-8h9q6\" (UID: \"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.631274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.631199 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9j8\" (UniqueName: \"kubernetes.io/projected/ff1e5356-333b-49bd-a169-6b2c5f603205-kube-api-access-dx9j8\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:29.631873 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.631856 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dp8\" (UniqueName: \"kubernetes.io/projected/af74fb5b-46b2-4082-a297-2707fa8f3fbe-kube-api-access-d7dp8\") pod \"managed-serviceaccount-addon-agent-69dd65c64d-mpj82\" (UID: \"af74fb5b-46b2-4082-a297-2707fa8f3fbe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.636331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.636310 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhjn\" (UniqueName: \"kubernetes.io/projected/3738bcd2-6eaa-4ba7-bc81-c1c27d594eda-kube-api-access-wkhjn\") pod \"cluster-proxy-proxy-agent-798574554f-7msh8\" (UID: \"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.693280 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.693241 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:48:29.696572 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.696540 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffd3795-b55e-4aa6-a848-934f0f73b203-cert\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.696727 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.696605 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dth6l\" (UniqueName: \"kubernetes.io/projected/fffd3795-b55e-4aa6-a848-934f0f73b203-kube-api-access-dth6l\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.699358 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.699326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffd3795-b55e-4aa6-a848-934f0f73b203-cert\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.708193 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.708166 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dth6l\" (UniqueName: \"kubernetes.io/projected/fffd3795-b55e-4aa6-a848-934f0f73b203-kube-api-access-dth6l\") pod \"ingress-canary-nn9qb\" (UID: \"fffd3795-b55e-4aa6-a848-934f0f73b203\") " pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.710002 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.709978 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" Jul 20 17:48:29.724327 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.724283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:29.741301 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.741255 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:29.777232 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.777198 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nn9qb" Jul 20 17:48:29.887780 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.887749 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82"] Jul 20 17:48:29.888941 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.888891 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8"] Jul 20 17:48:29.890561 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:29.890532 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf74fb5b_46b2_4082_a297_2707fa8f3fbe.slice/crio-79063d210354fc47fa5a2f161138880a2c6724a38068f589c013a7553d300955 WatchSource:0}: Error finding container 79063d210354fc47fa5a2f161138880a2c6724a38068f589c013a7553d300955: Status 404 returned error can't find the container with id 79063d210354fc47fa5a2f161138880a2c6724a38068f589c013a7553d300955 Jul 20 17:48:29.892153 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:29.892127 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3738bcd2_6eaa_4ba7_bc81_c1c27d594eda.slice/crio-d2bb601432b9b0592b96e77a6b13eaa5df47e4f89e1ee495d5a3c893c7bb2c06 WatchSource:0}: Error finding container d2bb601432b9b0592b96e77a6b13eaa5df47e4f89e1ee495d5a3c893c7bb2c06: Status 404 returned error can't find the container with id d2bb601432b9b0592b96e77a6b13eaa5df47e4f89e1ee495d5a3c893c7bb2c06 Jul 20 17:48:29.906499 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.906470 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6"] Jul 20 17:48:29.911691 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:29.911660 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1b9d20_2acf_4f1c_a45a_a4947dc8853b.slice/crio-454d8cbac7ea2fc5a0221037b0b5342a5eb884208b8b72952db1f5f3c56d6a96 WatchSource:0}: Error finding container 454d8cbac7ea2fc5a0221037b0b5342a5eb884208b8b72952db1f5f3c56d6a96: Status 404 returned error can't find the container with id 454d8cbac7ea2fc5a0221037b0b5342a5eb884208b8b72952db1f5f3c56d6a96 Jul 20 17:48:29.918504 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.918464 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:29.984650 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:29.984600 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nn9qb"] Jul 20 17:48:29.988754 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:29.988722 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffd3795_b55e_4aa6_a848_934f0f73b203.slice/crio-d808c4b8272a6473c7602931745b38f3b8cbe5d1d05b53fc49317223efc29bb3 WatchSource:0}: Error finding container d808c4b8272a6473c7602931745b38f3b8cbe5d1d05b53fc49317223efc29bb3: Status 404 returned error can't find the container with id d808c4b8272a6473c7602931745b38f3b8cbe5d1d05b53fc49317223efc29bb3 Jul 20 17:48:30.102004 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.101922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:30.102183 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.102065 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:30.102183 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:30.102091 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:30.102183 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:30.102154 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:31.10213643 +0000 UTC m=+39.040414723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:30.102358 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:30.102242 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:30.102358 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:30.102262 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:30.102358 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:30.102317 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:31.102298126 +0000 UTC m=+39.040576417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:30.105688 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.105653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn9qb" event={"ID":"fffd3795-b55e-4aa6-a848-934f0f73b203","Type":"ContainerStarted","Data":"d808c4b8272a6473c7602931745b38f3b8cbe5d1d05b53fc49317223efc29bb3"} Jul 20 17:48:30.107062 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.107028 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerStarted","Data":"d2bb601432b9b0592b96e77a6b13eaa5df47e4f89e1ee495d5a3c893c7bb2c06"} Jul 20 17:48:30.108118 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.108096 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" event={"ID":"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b","Type":"ContainerStarted","Data":"454d8cbac7ea2fc5a0221037b0b5342a5eb884208b8b72952db1f5f3c56d6a96"} Jul 20 17:48:30.109255 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:30.109235 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" event={"ID":"af74fb5b-46b2-4082-a297-2707fa8f3fbe","Type":"ContainerStarted","Data":"79063d210354fc47fa5a2f161138880a2c6724a38068f589c013a7553d300955"} Jul 20 17:48:31.110135 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:31.109921 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:31.110135 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:31.109972 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:31.110135 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:31.110138 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:31.111062 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:31.110156 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:31.111062 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:31.110214 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:33.110196362 +0000 UTC m=+41.048474631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:31.111062 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:31.110615 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:31.111062 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:31.110682 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:33.110666351 +0000 UTC m=+41.048944619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:33.130992 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:33.130878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:33.130992 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:33.130937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:33.131501 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:33.131123 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:33.131501 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:33.131195 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:37.131174171 +0000 UTC m=+45.069452442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:33.131501 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:33.131125 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:33.131501 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:33.131259 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:33.131501 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:33.131300 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:37.131288629 +0000 UTC m=+45.069566913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:34.092789 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:34.092748 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:34.317707 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:34.317662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:34.590130 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:34.590095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:34.810120 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:34.810072 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" limit=1024 Jul 20 17:48:37.164881 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:37.164837 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:37.164895 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:37.165007 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:37.165094 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:45.165071576 +0000 UTC m=+53.103349860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:37.165012 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:37.165126 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:37.165315 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:37.165172 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:48:45.165160129 +0000 UTC m=+53.103438400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:40.143290 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.143219 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerStarted","Data":"e4fdb52945b86be5a20a5904fb65a6a233ea9b41b83e8b99dd90b33035343efd"} Jul 20 17:48:40.145085 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.145046 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" event={"ID":"5e1b9d20-2acf-4f1c-a45a-a4947dc8853b","Type":"ContainerStarted","Data":"92700c4c48eedfa9933383754d77d1cc7f163ae4207f78333e816a88d7c74a2d"} Jul 20 17:48:40.145391 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.145371 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:40.147220 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.147191 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" event={"ID":"af74fb5b-46b2-4082-a297-2707fa8f3fbe","Type":"ContainerStarted","Data":"f225544ef1a0899806895123ea913f82094d853141971c7090b12f93d850d6cd"} Jul 20 17:48:40.147329 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.147304 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" Jul 20 17:48:40.151002 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.150977 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerStarted","Data":"5c794e5a8900366156c0fde0b9e3fc293b6b751a6f6f843b311146c45295b6d4"} Jul 20 17:48:40.152713 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.152690 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nn9qb" event={"ID":"fffd3795-b55e-4aa6-a848-934f0f73b203","Type":"ContainerStarted","Data":"5546dd2f85fd86f275a024d1641a190fc00218c778fd1f2984342c8cf1ebe03e"} Jul 20 17:48:40.195826 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.195778 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69dd65c64d-mpj82" podStartSLOduration=24.194075686 podStartE2EDuration="34.195760877s" totalImagesPullingTime="10.001685191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:40.194840644 +0000 UTC m=+48.133118934" watchObservedRunningTime="2026-07-20 17:48:40.195760877 +0000 UTC m=+48.134039188" Jul 20 17:48:40.195959 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.195891 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68cdcc9569-8h9q6" podStartSLOduration=24.211426121 podStartE2EDuration="34.195884864s" totalImagesPullingTime="9.984458743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:40.17103408 +0000 UTC m=+48.109312369" watchObservedRunningTime="2026-07-20 17:48:40.195884864 +0000 UTC m=+48.134163145" Jul 20 17:48:40.220792 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:40.220604 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nn9qb" podStartSLOduration=1.313337724 podStartE2EDuration="11.220583517s" totalImagesPullingTime="9.907245793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:40.219668298 +0000 UTC m=+48.157946615" watchObservedRunningTime="2026-07-20 17:48:40.220583517 +0000 UTC m=+48.158861806" Jul 20 17:48:41.157640 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:41.157586 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerStarted","Data":"1178603467d01b15a7469e2b0b5b3b02e1be7dcdd41cede0c777cae116d24c6c"} Jul 20 17:48:41.158068 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:41.157657 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerStarted","Data":"a134a6b57a01f332cceab2a915782a2d12d1b7f405cb6d2a0e7c4154e8e9d85f"} Jul 20 17:48:41.160203 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:41.160179 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="5c794e5a8900366156c0fde0b9e3fc293b6b751a6f6f843b311146c45295b6d4" exitCode=0 Jul 20 17:48:41.160336 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:41.160272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"5c794e5a8900366156c0fde0b9e3fc293b6b751a6f6f843b311146c45295b6d4"} Jul 20 17:48:41.194986 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:41.194889 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" podStartSLOduration=25.179206337 podStartE2EDuration="35.1948723s" totalImagesPullingTime="10.015665963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:41.193762984 +0000 UTC m=+49.132041320" watchObservedRunningTime="2026-07-20 17:48:41.1948723 +0000 UTC m=+49.133150584" Jul 20 17:48:42.166140 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:42.166103 2554 generic.go:356] "Generic (PLEG): container finished" podID="fa6ae863-c428-499f-8d98-a2a8b36edfeb" containerID="2f55400f3b4f6ee4e77a90c53c421878b717450052f2513258203227dc5affc9" exitCode=0 Jul 20 17:48:42.166587 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:42.166186 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerDied","Data":"2f55400f3b4f6ee4e77a90c53c421878b717450052f2513258203227dc5affc9"} Jul 20 17:48:43.172131 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:43.172092 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" event={"ID":"fa6ae863-c428-499f-8d98-a2a8b36edfeb","Type":"ContainerStarted","Data":"9888ff984c6028fd164d7ed0ca487bf9c00ed9957a04ea56ce95c3e3efe49839"} Jul 20 17:48:43.211609 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:43.211548 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9r2vz" podStartSLOduration=9.328676772 podStartE2EDuration="49.211526489s" totalImagesPullingTime="39.882849717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:48:43.208852716 +0000 UTC m=+51.147131025" watchObservedRunningTime="2026-07-20 17:48:43.211526489 +0000 UTC m=+51.149804780" Jul 20 17:48:45.233970 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:45.233933 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:48:45.233970 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:45.233970 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:48:45.234388 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:45.234075 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:48:45.234388 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:45.234078 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:48:45.234388 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:45.234150 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:01.234133842 +0000 UTC m=+69.172412115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:48:45.234388 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:45.234086 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:48:45.234388 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:45.234214 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:01.234203694 +0000 UTC m=+69.172481961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:48:54.095412 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:54.095381 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k7dh8" Jul 20 17:48:58.833509 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.833462 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:58.833972 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.833527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:48:58.836497 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.836472 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 17:48:58.836612 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.836476 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 17:48:58.844527 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:58.844506 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 17:48:58.844608 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:48:58.844573 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs podName:084c96d0-3337-4ff6-8122-73354392e16c nodeName:}" failed. No retries permitted until 2026-07-20 17:50:02.844555592 +0000 UTC m=+130.782833860 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs") pod "network-metrics-daemon-46fm2" (UID: "084c96d0-3337-4ff6-8122-73354392e16c") : secret "metrics-daemon-secret" not found Jul 20 17:48:58.847661 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.847639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48680f07-ab02-4e7f-8416-949da11961ba-original-pull-secret\") pod \"global-pull-secret-syncer-mtfkc\" (UID: \"48680f07-ab02-4e7f-8416-949da11961ba\") " pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:58.934274 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.934231 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:58.937745 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.937725 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 17:48:58.947999 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.947976 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 17:48:58.958071 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.958044 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw58\" (UniqueName: \"kubernetes.io/projected/034807dc-20ae-4b6a-8ed6-277f795efba1-kube-api-access-hlw58\") pod \"network-check-target-dttfw\" (UID: \"034807dc-20ae-4b6a-8ed6-277f795efba1\") " pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:58.968914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:58.968887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mtfkc" Jul 20 17:48:59.092970 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:59.092888 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mtfkc"] Jul 20 17:48:59.095973 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:59.095943 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48680f07_ab02_4e7f_8416_949da11961ba.slice/crio-e4359625b6f5697f09ac36fc3ed6bd8f3e93d9988c43b73b202345d4aec1c1f0 WatchSource:0}: Error finding container e4359625b6f5697f09ac36fc3ed6bd8f3e93d9988c43b73b202345d4aec1c1f0: Status 404 returned error can't find the container with id e4359625b6f5697f09ac36fc3ed6bd8f3e93d9988c43b73b202345d4aec1c1f0 Jul 20 17:48:59.219473 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:59.219428 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtfkc" event={"ID":"48680f07-ab02-4e7f-8416-949da11961ba","Type":"ContainerStarted","Data":"e4359625b6f5697f09ac36fc3ed6bd8f3e93d9988c43b73b202345d4aec1c1f0"} Jul 20 17:48:59.261131 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:59.261107 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcx42\"" Jul 20 17:48:59.268894 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:59.268871 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:48:59.400478 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:48:59.400446 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dttfw"] Jul 20 17:48:59.403095 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:48:59.403067 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034807dc_20ae_4b6a_8ed6_277f795efba1.slice/crio-5999bc0034a3c9a6fa95bf868b1438c57c01338e8b771ca5e6b461395f1360e5 WatchSource:0}: Error finding container 5999bc0034a3c9a6fa95bf868b1438c57c01338e8b771ca5e6b461395f1360e5: Status 404 returned error can't find the container with id 5999bc0034a3c9a6fa95bf868b1438c57c01338e8b771ca5e6b461395f1360e5 Jul 20 17:49:00.225764 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:00.225721 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dttfw" event={"ID":"034807dc-20ae-4b6a-8ed6-277f795efba1","Type":"ContainerStarted","Data":"5999bc0034a3c9a6fa95bf868b1438c57c01338e8b771ca5e6b461395f1360e5"} Jul 20 17:49:01.250309 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:01.250258 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:01.250317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:01.250440 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:01.250479 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:01.250495 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:01.250516 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:33.25049561 +0000 UTC m=+101.188773900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:49:01.250827 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:01.250545 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:49:33.250527829 +0000 UTC m=+101.188806102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:49:05.243754 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:05.243721 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dttfw" event={"ID":"034807dc-20ae-4b6a-8ed6-277f795efba1","Type":"ContainerStarted","Data":"e4ef19d973f37355eaf71df1c11b6efe7e2cd3bfb9eb21d4ce6e7023e12e5af1"} Jul 20 17:49:05.244300 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:05.243855 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:49:05.245242 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:05.245219 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mtfkc" event={"ID":"48680f07-ab02-4e7f-8416-949da11961ba","Type":"ContainerStarted","Data":"3cdf27c3baa1c7f2f67e488060195e076627697256a9ef3fdd750c09260751c4"} Jul 20 17:49:05.264963 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:05.264906 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dttfw" podStartSLOduration=66.414545261 podStartE2EDuration="1m11.264891301s" totalImagesPullingTime="4.85034604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:05.262804572 +0000 UTC m=+73.201082885" watchObservedRunningTime="2026-07-20 17:49:05.264891301 +0000 UTC m=+73.203169582" Jul 20 17:49:33.297271 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:33.297232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:33.297279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") pod \"image-registry-757ff4758d-ql765\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:33.297392 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:33.297468 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls podName:ff1e5356-333b-49bd-a169-6b2c5f603205 nodeName:}" failed. No retries permitted until 2026-07-20 17:50:37.297452005 +0000 UTC m=+165.235730273 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls") pod "dns-default-t54gs" (UID: "ff1e5356-333b-49bd-a169-6b2c5f603205") : secret "dns-default-metrics-tls" not found Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:33.297395 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:33.297493 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757ff4758d-ql765: secret "image-registry-tls" not found Jul 20 17:49:33.297723 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:33.297551 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls podName:db83a40b-dc92-4cd2-ae34-931b4fb5a0b5 nodeName:}" failed. No retries permitted until 2026-07-20 17:50:37.29753707 +0000 UTC m=+165.235815347 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls") pod "image-registry-757ff4758d-ql765" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5") : secret "image-registry-tls" not found Jul 20 17:49:34.459193 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.459141 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mtfkc" podStartSLOduration=95.296999245 podStartE2EDuration="1m40.45912137s" totalImagesPullingTime="5.162122125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:05.287698517 +0000 UTC m=+73.225976828" watchObservedRunningTime="2026-07-20 17:49:34.45912137 +0000 UTC m=+102.397399638" Jul 20 17:49:34.459770 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.459753 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gxtv7"] Jul 20 17:49:34.464282 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.464266 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.470037 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.470009 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hq7ht\"" Jul 20 17:49:34.470486 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.470264 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 17:49:34.471140 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.471116 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 17:49:34.471320 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.471117 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 17:49:34.471320 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.471282 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 17:49:34.499143 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.499072 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gxtv7"] Jul 20 17:49:34.606009 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.605972 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61b66212-57ef-433f-acd2-d05cd43bd46c-crio-socket\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.606209 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.606030 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.606209 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.606150 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.606332 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.606206 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b66212-57ef-433f-acd2-d05cd43bd46c-data-volume\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.606332 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.606233 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdd7\" (UniqueName: \"kubernetes.io/projected/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-api-access-9tdd7\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707604 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707560 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707604 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707609 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b66212-57ef-433f-acd2-d05cd43bd46c-data-volume\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdd7\" (UniqueName: \"kubernetes.io/projected/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-api-access-9tdd7\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707700 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61b66212-57ef-433f-acd2-d05cd43bd46c-crio-socket\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:34.707708 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707736 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:34.707779 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:35.207763801 +0000 UTC m=+103.146042068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : configmap "kube-rbac-proxy" not found Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707816 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61b66212-57ef-433f-acd2-d05cd43bd46c-crio-socket\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:34.707827 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:34.707898 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:34.707894 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:35.207878881 +0000 UTC m=+103.146157158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:34.708223 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.707988 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b66212-57ef-433f-acd2-d05cd43bd46c-data-volume\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:34.721424 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:34.721367 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdd7\" (UniqueName: \"kubernetes.io/projected/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-api-access-9tdd7\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:35.211422 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:35.211381 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:35.211609 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:35.211433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:35.211609 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:35.211525 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:35.211609 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:35.211549 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:35.211609 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:35.211599 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:36.211579885 +0000 UTC m=+104.149858154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:35.211817 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:35.211643 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:36.211610257 +0000 UTC m=+104.149888533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : configmap "kube-rbac-proxy" not found Jul 20 17:49:36.219180 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:36.219137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:36.219564 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:36.219229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:36.219564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:36.219256 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:36.219564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:36.219324 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:36.219564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:36.219327 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:38.219311126 +0000 UTC m=+106.157589393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : configmap "kube-rbac-proxy" not found Jul 20 17:49:36.219564 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:36.219370 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:38.219359484 +0000 UTC m=+106.157637751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:36.253758 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:36.253720 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dttfw" Jul 20 17:49:36.295120 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:36.295087 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5l5wk_d7dff9a9-ab37-4319-bebc-799e402d6e2c/dns-node-resolver/0.log" Jul 20 17:49:37.295305 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:37.295278 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-78xsn_baa865ba-f651-4070-8e45-026a0bda5a09/node-ca/0.log" Jul 20 17:49:37.954445 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:37.954415 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz"] Jul 20 17:49:37.983351 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:37.983326 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz"] Jul 20 17:49:37.983480 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:37.983416 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" Jul 20 17:49:37.987871 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:37.987847 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pmgxm\"" Jul 20 17:49:38.133415 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.133378 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmxd\" (UniqueName: \"kubernetes.io/projected/a4be5e69-8477-4778-a249-9a05279c2f55-kube-api-access-wbmxd\") pod \"network-check-source-7bc5b6bf85-ndbkz\" (UID: \"a4be5e69-8477-4778-a249-9a05279c2f55\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" Jul 20 17:49:38.233817 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.233734 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:38.233817 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.233778 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmxd\" (UniqueName: \"kubernetes.io/projected/a4be5e69-8477-4778-a249-9a05279c2f55-kube-api-access-wbmxd\") pod \"network-check-source-7bc5b6bf85-ndbkz\" (UID: \"a4be5e69-8477-4778-a249-9a05279c2f55\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" Jul 20 17:49:38.233990 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:38.233863 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:38.233990 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:38.233939 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:42.233922863 +0000 UTC m=+110.172201148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : configmap "kube-rbac-proxy" not found Jul 20 17:49:38.234074 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.233983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:38.234113 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:38.234088 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:38.234150 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:38.234122 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:42.234114254 +0000 UTC m=+110.172392522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:38.242802 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.242776 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmxd\" (UniqueName: \"kubernetes.io/projected/a4be5e69-8477-4778-a249-9a05279c2f55-kube-api-access-wbmxd\") pod \"network-check-source-7bc5b6bf85-ndbkz\" (UID: \"a4be5e69-8477-4778-a249-9a05279c2f55\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" Jul 20 17:49:38.292326 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.292283 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" Jul 20 17:49:38.441931 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.441899 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz"] Jul 20 17:49:38.445521 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:49:38.445494 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4be5e69_8477_4778_a249_9a05279c2f55.slice/crio-f2dbfe2f3f5d7512bad5914ccfa75dd305c4351c1be10f4226c633889d7daa0a WatchSource:0}: Error finding container f2dbfe2f3f5d7512bad5914ccfa75dd305c4351c1be10f4226c633889d7daa0a: Status 404 returned error can't find the container with id f2dbfe2f3f5d7512bad5914ccfa75dd305c4351c1be10f4226c633889d7daa0a Jul 20 17:49:38.489859 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:38.489836 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn9qb_fffd3795-b55e-4aa6-a848-934f0f73b203/serve-healthcheck-canary/0.log" Jul 20 17:49:39.348381 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:39.348346 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" event={"ID":"a4be5e69-8477-4778-a249-9a05279c2f55","Type":"ContainerStarted","Data":"47254a2c0b2b063513035e1065c8ea92ed7d78ffd511a561babde49c7f4bc082"} Jul 20 17:49:39.348583 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:39.348396 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" event={"ID":"a4be5e69-8477-4778-a249-9a05279c2f55","Type":"ContainerStarted","Data":"f2dbfe2f3f5d7512bad5914ccfa75dd305c4351c1be10f4226c633889d7daa0a"} Jul 20 17:49:39.379297 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:39.379254 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ndbkz" podStartSLOduration=2.379237627 podStartE2EDuration="2.379237627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:49:39.377841541 +0000 UTC m=+107.316119831" watchObservedRunningTime="2026-07-20 17:49:39.379237627 +0000 UTC m=+107.317515916" Jul 20 17:49:42.265456 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:42.265416 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:42.265857 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:42.265468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:42.265857 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:42.265558 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 17:49:42.265857 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:42.265560 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:42.265857 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:42.265617 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:50.265602909 +0000 UTC m=+118.203881177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : configmap "kube-rbac-proxy" not found Jul 20 17:49:42.265857 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:42.265661 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:49:50.265642904 +0000 UTC m=+118.203921185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:50.328403 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:50.328356 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:50.328979 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:50.328468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:50.328979 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:50.328581 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 17:49:50.328979 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:49:50.328663 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls podName:61b66212-57ef-433f-acd2-d05cd43bd46c nodeName:}" failed. No retries permitted until 2026-07-20 17:50:06.328622183 +0000 UTC m=+134.266900474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gxtv7" (UID: "61b66212-57ef-433f-acd2-d05cd43bd46c") : secret "insights-runtime-extractor-tls" not found Jul 20 17:49:50.329147 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:50.329078 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61b66212-57ef-433f-acd2-d05cd43bd46c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:49:59.696029 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:49:59.695978 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" podUID="3738bcd2-6eaa-4ba7-bc81-c1c27d594eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 17:50:02.932995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:02.932943 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:50:02.935228 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:02.935204 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084c96d0-3337-4ff6-8122-73354392e16c-metrics-certs\") pod \"network-metrics-daemon-46fm2\" (UID: \"084c96d0-3337-4ff6-8122-73354392e16c\") " pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:50:03.170297 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:03.170268 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mjfj5\"" Jul 20 17:50:03.174687 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:03.174667 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46fm2" Jul 20 17:50:03.318515 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:50:03.318482 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084c96d0_3337_4ff6_8122_73354392e16c.slice/crio-96e0a6f3785e9412f09b5e7f9779fe2e58397c0d9e9d1cae13f63bd50aee52c5 WatchSource:0}: Error finding container 96e0a6f3785e9412f09b5e7f9779fe2e58397c0d9e9d1cae13f63bd50aee52c5: Status 404 returned error can't find the container with id 96e0a6f3785e9412f09b5e7f9779fe2e58397c0d9e9d1cae13f63bd50aee52c5 Jul 20 17:50:03.318822 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:03.318800 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46fm2"] Jul 20 17:50:03.424122 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:03.424078 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46fm2" event={"ID":"084c96d0-3337-4ff6-8122-73354392e16c","Type":"ContainerStarted","Data":"96e0a6f3785e9412f09b5e7f9779fe2e58397c0d9e9d1cae13f63bd50aee52c5"} Jul 20 17:50:05.431987 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:05.431896 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46fm2" event={"ID":"084c96d0-3337-4ff6-8122-73354392e16c","Type":"ContainerStarted","Data":"cdab89c3568cdbccb0b47badd9442d50f8f70c155894b73f843fb61bc500a522"} Jul 20 17:50:05.431987 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:05.431959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46fm2" event={"ID":"084c96d0-3337-4ff6-8122-73354392e16c","Type":"ContainerStarted","Data":"9ec1461651cf6de44ed6323ef05b1c73220303f3cd59415acb5d5a78f83997c7"} Jul 20 17:50:05.457788 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:05.457736 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-46fm2" podStartSLOduration=129.615641916 podStartE2EDuration="2m11.457721139s" totalImagesPullingTime="1.842079223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:05.456766869 +0000 UTC m=+133.395045160" watchObservedRunningTime="2026-07-20 17:50:05.457721139 +0000 UTC m=+133.395999429" Jul 20 17:50:06.359605 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:06.359570 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:50:06.361817 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:06.361785 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61b66212-57ef-433f-acd2-d05cd43bd46c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gxtv7\" (UID: \"61b66212-57ef-433f-acd2-d05cd43bd46c\") " pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:50:06.576187 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:06.576162 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hq7ht\"" Jul 20 17:50:06.583437 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:06.583417 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gxtv7" Jul 20 17:50:06.710336 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:06.710312 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gxtv7"] Jul 20 17:50:06.712511 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:50:06.712490 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b66212_57ef_433f_acd2_d05cd43bd46c.slice/crio-c16429504d54375d3bbcdea0a05654a0cb7ed35b8d4dcca1a8d57d718ff8bcf3 WatchSource:0}: Error finding container c16429504d54375d3bbcdea0a05654a0cb7ed35b8d4dcca1a8d57d718ff8bcf3: Status 404 returned error can't find the container with id c16429504d54375d3bbcdea0a05654a0cb7ed35b8d4dcca1a8d57d718ff8bcf3 Jul 20 17:50:07.440228 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:07.440140 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxtv7" event={"ID":"61b66212-57ef-433f-acd2-d05cd43bd46c","Type":"ContainerStarted","Data":"c16429504d54375d3bbcdea0a05654a0cb7ed35b8d4dcca1a8d57d718ff8bcf3"} Jul 20 17:50:09.695168 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:09.695125 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" podUID="3738bcd2-6eaa-4ba7-bc81-c1c27d594eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 17:50:12.461781 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:12.461741 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxtv7" event={"ID":"61b66212-57ef-433f-acd2-d05cd43bd46c","Type":"ContainerStarted","Data":"abf6a6604607da9da4c4edce13b8bf11397b36c941d0428d76150f5a75e735f3"} Jul 20 17:50:15.473995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:15.473957 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxtv7" event={"ID":"61b66212-57ef-433f-acd2-d05cd43bd46c","Type":"ContainerStarted","Data":"90f7d31e59d14e2b2353b7bcbff1d483b6bb4430a9b071649b618a3a03fad10d"} Jul 20 17:50:17.137217 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.137177 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n8xwb"] Jul 20 17:50:17.140503 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.140483 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.151292 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151270 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 17:50:17.151400 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151329 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 17:50:17.151400 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151332 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 17:50:17.151906 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151888 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 17:50:17.151906 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151901 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 17:50:17.152049 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.151912 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dmjq\"" Jul 20 17:50:17.184313 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.184294 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 17:50:17.241451 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241425 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241591 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241472 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-metrics-client-ca\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241591 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241557 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-root\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241713 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241650 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241713 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241696 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-sys\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241798 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241715 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-tls\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241798 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzsd\" (UniqueName: \"kubernetes.io/projected/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-kube-api-access-vlzsd\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241798 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241776 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-textfile\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.241916 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.241807 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-wtmp\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342780 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342747 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-root\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342813 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-sys\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342830 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-tls\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342845 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzsd\" (UniqueName: \"kubernetes.io/projected/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-kube-api-access-vlzsd\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-textfile\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342869 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-root\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-wtmp\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.342939 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.342914 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-sys\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343401 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343005 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343401 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343014 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-wtmp\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343401 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343048 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-metrics-client-ca\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343552 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343420 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-textfile\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343834 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343805 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.343955 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.343814 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-metrics-client-ca\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.345317 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.345294 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.345460 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.345441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-node-exporter-tls\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.412554 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.412475 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzsd\" (UniqueName: \"kubernetes.io/projected/af3affcc-e4a1-4fb3-9e97-8bb91700d6f6-kube-api-access-vlzsd\") pod \"node-exporter-n8xwb\" (UID: \"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6\") " pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:17.451308 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:17.451275 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n8xwb" Jul 20 17:50:18.943101 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:50:18.943066 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3affcc_e4a1_4fb3_9e97_8bb91700d6f6.slice/crio-bb8f2b2e93bd4b2d3b97b3a93421554f19027aa3f27ef65e2b6c4c8c79866d4a WatchSource:0}: Error finding container bb8f2b2e93bd4b2d3b97b3a93421554f19027aa3f27ef65e2b6c4c8c79866d4a: Status 404 returned error can't find the container with id bb8f2b2e93bd4b2d3b97b3a93421554f19027aa3f27ef65e2b6c4c8c79866d4a Jul 20 17:50:19.496570 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.496515 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gxtv7" event={"ID":"61b66212-57ef-433f-acd2-d05cd43bd46c","Type":"ContainerStarted","Data":"a40260a1b287dd7a5a4e3b17100083174e4435c7c7c6f2bc20b7e1b4f5856724"} Jul 20 17:50:19.497773 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.497739 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8xwb" event={"ID":"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6","Type":"ContainerStarted","Data":"bb8f2b2e93bd4b2d3b97b3a93421554f19027aa3f27ef65e2b6c4c8c79866d4a"} Jul 20 17:50:19.539759 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.539715 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gxtv7" podStartSLOduration=33.381775422 podStartE2EDuration="45.539696127s" totalImagesPullingTime="12.157920705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:49:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:19.539309125 +0000 UTC m=+147.477587415" watchObservedRunningTime="2026-07-20 17:50:19.539696127 +0000 UTC m=+147.477974622" Jul 20 17:50:19.694863 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.694824 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" podUID="3738bcd2-6eaa-4ba7-bc81-c1c27d594eda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 17:50:19.695025 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.694904 2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" Jul 20 17:50:19.695492 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.695448 2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1178603467d01b15a7469e2b0b5b3b02e1be7dcdd41cede0c777cae116d24c6c"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 20 17:50:19.695598 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:19.695516 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" podUID="3738bcd2-6eaa-4ba7-bc81-c1c27d594eda" containerName="service-proxy" containerID="cri-o://1178603467d01b15a7469e2b0b5b3b02e1be7dcdd41cede0c777cae116d24c6c" gracePeriod=30 Jul 20 17:50:20.502401 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:20.502370 2554 generic.go:356] "Generic (PLEG): container finished" podID="af3affcc-e4a1-4fb3-9e97-8bb91700d6f6" containerID="ccc1832f8d521eba4d8e0ca3a50fb444cc530741d4fe9e84b68b9d2039d23e30" exitCode=0 Jul 20 17:50:20.502816 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:20.502462 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8xwb" event={"ID":"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6","Type":"ContainerDied","Data":"ccc1832f8d521eba4d8e0ca3a50fb444cc530741d4fe9e84b68b9d2039d23e30"} Jul 20 17:50:20.504876 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:20.504854 2554 generic.go:356] "Generic (PLEG): container finished" podID="3738bcd2-6eaa-4ba7-bc81-c1c27d594eda" containerID="1178603467d01b15a7469e2b0b5b3b02e1be7dcdd41cede0c777cae116d24c6c" exitCode=2 Jul 20 17:50:20.504993 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:20.504926 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerDied","Data":"1178603467d01b15a7469e2b0b5b3b02e1be7dcdd41cede0c777cae116d24c6c"} Jul 20 17:50:20.504993 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:20.504966 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798574554f-7msh8" event={"ID":"3738bcd2-6eaa-4ba7-bc81-c1c27d594eda","Type":"ContainerStarted","Data":"bbf0a60dfc8dfbfd1edf6e15741281be548a2ca52f47869853891e36aa0efd96"} Jul 20 17:50:21.510417 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:21.510384 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8xwb" event={"ID":"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6","Type":"ContainerStarted","Data":"ad69b45ebdcfe397415727f628fc5ccc207600746b23ce4c907ce4f80e21a753"} Jul 20 17:50:21.510789 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:21.510435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n8xwb" event={"ID":"af3affcc-e4a1-4fb3-9e97-8bb91700d6f6","Type":"ContainerStarted","Data":"b7cf3c9b3b636f2b121ba2dbc2e197a3769de8a7f90c3992ecd88e1b35d09a95"} Jul 20 17:50:21.567221 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:21.567178 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n8xwb" podStartSLOduration=3.443071842 podStartE2EDuration="4.567163294s" totalImagesPullingTime="1.061690396s" totalInitContainerRuntime="62.401056ms" isStatefulPod=true podCreationTimestamp="2026-07-20 17:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:21.565644831 +0000 UTC m=+149.503923146" watchObservedRunningTime="2026-07-20 17:50:21.567163294 +0000 UTC m=+149.505441583" Jul 20 17:50:25.700593 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:25.700551 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757ff4758d-ql765"] Jul 20 17:50:25.700978 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:50:25.700781 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-757ff4758d-ql765" podUID="db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" Jul 20 17:50:26.525829 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.525798 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:50:26.532117 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.532097 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608115 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608179 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608225 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608286 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608312 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zr69\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608340 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.608613 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608364 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates\") pod \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\" (UID: \"db83a40b-dc92-4cd2-ae34-931b4fb5a0b5\") " Jul 20 17:50:26.609102 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608780 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 17:50:26.609102 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.608891 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:50:26.609208 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.609116 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 17:50:26.610941 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.610913 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69" (OuterVolumeSpecName: "kube-api-access-6zr69") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "kube-api-access-6zr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:50:26.611176 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.611151 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:50:26.611238 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.611179 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:50:26.611238 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.611225 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" (UID: "db83a40b-dc92-4cd2-ae34-931b4fb5a0b5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:50:26.709394 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709360 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-image-registry-private-configuration\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709394 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709388 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-bound-sa-token\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709394 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709399 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr69\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-kube-api-access-6zr69\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709847 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709407 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-ca-trust-extracted\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709847 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709416 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-certificates\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709847 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709424 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-installation-pull-secrets\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:26.709847 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:26.709433 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-trusted-ca\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:27.528806 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:27.528777 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757ff4758d-ql765" Jul 20 17:50:27.568187 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:27.568160 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757ff4758d-ql765"] Jul 20 17:50:27.571006 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:27.570983 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-757ff4758d-ql765"] Jul 20 17:50:27.614452 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:27.614425 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5-registry-tls\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:50:28.547740 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:28.547706 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db83a40b-dc92-4cd2-ae34-931b4fb5a0b5" path="/var/lib/kubelet/pods/db83a40b-dc92-4cd2-ae34-931b4fb5a0b5/volumes" Jul 20 17:50:32.432969 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:50:32.432933 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-t54gs" podUID="ff1e5356-333b-49bd-a169-6b2c5f603205" Jul 20 17:50:32.545306 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:32.545262 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t54gs" Jul 20 17:50:37.386700 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:37.386660 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:50:37.388933 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:37.388911 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff1e5356-333b-49bd-a169-6b2c5f603205-metrics-tls\") pod \"dns-default-t54gs\" (UID: \"ff1e5356-333b-49bd-a169-6b2c5f603205\") " pod="openshift-dns/dns-default-t54gs" Jul 20 17:50:37.648863 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:37.648784 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkvwc\"" Jul 20 17:50:37.656359 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:37.656334 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t54gs" Jul 20 17:50:37.989914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:37.989883 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t54gs"] Jul 20 17:50:37.991478 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:50:37.991441 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1e5356_333b_49bd_a169_6b2c5f603205.slice/crio-964168dfccffa6a4c5397b7442db2050341ac7b1f2ca534a71b227e2cf807a9e WatchSource:0}: Error finding container 964168dfccffa6a4c5397b7442db2050341ac7b1f2ca534a71b227e2cf807a9e: Status 404 returned error can't find the container with id 964168dfccffa6a4c5397b7442db2050341ac7b1f2ca534a71b227e2cf807a9e Jul 20 17:50:38.566914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:38.566874 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t54gs" event={"ID":"ff1e5356-333b-49bd-a169-6b2c5f603205","Type":"ContainerStarted","Data":"964168dfccffa6a4c5397b7442db2050341ac7b1f2ca534a71b227e2cf807a9e"} Jul 20 17:50:41.577885 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:41.577848 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t54gs" event={"ID":"ff1e5356-333b-49bd-a169-6b2c5f603205","Type":"ContainerStarted","Data":"e643b902d9039cd57a21d5d5639fe5c6666c5921217d85243e87778491e167fa"} Jul 20 17:50:41.578339 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:41.577902 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t54gs" event={"ID":"ff1e5356-333b-49bd-a169-6b2c5f603205","Type":"ContainerStarted","Data":"730e100dc157ba6a751acd83b2fc0c669c4614f033f73ba818643a721e75ca38"} Jul 20 17:50:41.578339 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:41.577981 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t54gs" Jul 20 17:50:41.606284 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:41.606222 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t54gs" podStartSLOduration=129.818395809 podStartE2EDuration="2m12.606206326s" totalImagesPullingTime="2.787810517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:50:41.605794994 +0000 UTC m=+169.544073295" watchObservedRunningTime="2026-07-20 17:50:41.606206326 +0000 UTC m=+169.544484617" Jul 20 17:50:51.582768 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:50:51.582692 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t54gs" Jul 20 17:52:52.418210 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:52:52.418182 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:52:52.422848 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:52:52.422822 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:52:52.426603 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:52:52.426581 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 17:57:52.439187 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:57:52.439160 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:57:52.440845 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:57:52.440823 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 17:58:32.492811 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.492777 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l"] Jul 20 17:58:32.495598 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.495582 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.498706 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.498682 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 17:58:32.498820 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.498739 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 17:58:32.499919 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.499905 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6qmxk\"" Jul 20 17:58:32.507263 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.507243 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l"] Jul 20 17:58:32.515386 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.515362 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjbl\" (UniqueName: \"kubernetes.io/projected/db393466-012a-461f-be36-f39b92fcd94d-kube-api-access-pjjbl\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.515468 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.515402 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db393466-012a-461f-be36-f39b92fcd94d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.616581 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.616525 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db393466-012a-461f-be36-f39b92fcd94d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.616787 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.616615 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjbl\" (UniqueName: \"kubernetes.io/projected/db393466-012a-461f-be36-f39b92fcd94d-kube-api-access-pjjbl\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.616944 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.616925 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db393466-012a-461f-be36-f39b92fcd94d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.626983 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.626955 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjbl\" (UniqueName: \"kubernetes.io/projected/db393466-012a-461f-be36-f39b92fcd94d-kube-api-access-pjjbl\") pod \"openshift-lws-operator-fd8ccff4c-zvw7l\" (UID: \"db393466-012a-461f-be36-f39b92fcd94d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.804225 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.804123 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" Jul 20 17:58:32.931940 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.931915 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l"] Jul 20 17:58:32.934260 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:58:32.934237 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb393466_012a_461f_be36_f39b92fcd94d.slice/crio-c9aeebbb710fffaac8238c2e15e5b68b4729543f8442f5c60c8b7a5d8476c1ea WatchSource:0}: Error finding container c9aeebbb710fffaac8238c2e15e5b68b4729543f8442f5c60c8b7a5d8476c1ea: Status 404 returned error can't find the container with id c9aeebbb710fffaac8238c2e15e5b68b4729543f8442f5c60c8b7a5d8476c1ea Jul 20 17:58:32.935732 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:32.935712 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 17:58:33.079151 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:33.079071 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" event={"ID":"db393466-012a-461f-be36-f39b92fcd94d","Type":"ContainerStarted","Data":"c9aeebbb710fffaac8238c2e15e5b68b4729543f8442f5c60c8b7a5d8476c1ea"} Jul 20 17:58:36.092886 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:36.092851 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" event={"ID":"db393466-012a-461f-be36-f39b92fcd94d","Type":"ContainerStarted","Data":"3795eea4bb7d3adc8d79b8e2b744dc3cb5c557bbfb8e82a848d32c7ea6418ea3"} Jul 20 17:58:36.111330 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:36.111277 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zvw7l" podStartSLOduration=1.4346062179999999 podStartE2EDuration="4.111262679s" totalImagesPullingTime="2.676656461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:36.109408534 +0000 UTC m=+644.047686843" watchObservedRunningTime="2026-07-20 17:58:36.111262679 +0000 UTC m=+644.049540969" Jul 20 17:58:52.960021 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:52.959986 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv"] Jul 20 17:58:52.966798 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:52.966773 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:52.976083 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:52.976062 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 17:58:52.989238 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:52.989213 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 17:58:53.016752 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.016721 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jtcl9\"" Jul 20 17:58:53.016896 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.016768 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 17:58:53.016896 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.016721 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 17:58:53.017855 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.017837 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv"] Jul 20 17:58:53.062573 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.062537 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.062573 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.062571 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.062839 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.062596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcsk\" (UniqueName: \"kubernetes.io/projected/f753c167-f037-420a-a79b-1843f6f5adf2-kube-api-access-fzcsk\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.062839 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.062707 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.062839 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.062728 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.163902 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.163874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.163902 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.163902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.164137 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.163926 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcsk\" (UniqueName: \"kubernetes.io/projected/f753c167-f037-420a-a79b-1843f6f5adf2-kube-api-access-fzcsk\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.164137 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.163959 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.164137 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.163974 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.164328 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.164271 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-manifests\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.166373 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.166339 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f753c167-f037-420a-a79b-1843f6f5adf2-tmp\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.166470 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.166433 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.166470 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.166454 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f753c167-f037-420a-a79b-1843f6f5adf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.177312 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.177288 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcsk\" (UniqueName: \"kubernetes.io/projected/f753c167-f037-420a-a79b-1843f6f5adf2-kube-api-access-fzcsk\") pod \"opendatahub-operator-controller-manager-7f8f8bd966-mnlgv\" (UID: \"f753c167-f037-420a-a79b-1843f6f5adf2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.276318 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.276287 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:53.445169 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:53.441834 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv"] Jul 20 17:58:53.445380 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:58:53.445350 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf753c167_f037_420a_a79b_1843f6f5adf2.slice/crio-19229894f53a734675be1af9aa2087fdde680f85f6e8c91a715a93ba29d4bfee WatchSource:0}: Error finding container 19229894f53a734675be1af9aa2087fdde680f85f6e8c91a715a93ba29d4bfee: Status 404 returned error can't find the container with id 19229894f53a734675be1af9aa2087fdde680f85f6e8c91a715a93ba29d4bfee Jul 20 17:58:54.156403 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:54.156369 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" event={"ID":"f753c167-f037-420a-a79b-1843f6f5adf2","Type":"ContainerStarted","Data":"19229894f53a734675be1af9aa2087fdde680f85f6e8c91a715a93ba29d4bfee"} Jul 20 17:58:57.169899 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:57.169855 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" event={"ID":"f753c167-f037-420a-a79b-1843f6f5adf2","Type":"ContainerStarted","Data":"b036baf0f4e598120ed20c5adc0e83a002c90d7f9cba4f671bc42652c4503d96"} Jul 20 17:58:58.174894 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:58.174859 2554 generic.go:356] "Generic (PLEG): container finished" podID="f753c167-f037-420a-a79b-1843f6f5adf2" containerID="b036baf0f4e598120ed20c5adc0e83a002c90d7f9cba4f671bc42652c4503d96" exitCode=0 Jul 20 17:58:58.175305 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:58.174945 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" event={"ID":"f753c167-f037-420a-a79b-1843f6f5adf2","Type":"ContainerDied","Data":"b036baf0f4e598120ed20c5adc0e83a002c90d7f9cba4f671bc42652c4503d96"} Jul 20 17:58:59.180149 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:59.180114 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" event={"ID":"f753c167-f037-420a-a79b-1843f6f5adf2","Type":"ContainerStarted","Data":"c2e23068cd9dc138e163bcea97e28b22fb386b162eb84c76e78bf0170c0ac5e5"} Jul 20 17:58:59.180533 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:59.180224 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:58:59.208722 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:58:59.208649 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" podStartSLOduration=2.79741184 podStartE2EDuration="7.208618375s" totalImagesPullingTime="3.32033589s" totalInitContainerRuntime="1.090870645s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:59.207548587 +0000 UTC m=+667.145826898" watchObservedRunningTime="2026-07-20 17:58:59.208618375 +0000 UTC m=+667.146896665" Jul 20 17:59:10.186102 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:10.186074 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f8f8bd966-mnlgv" Jul 20 17:59:12.858308 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:12.858273 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75cmz"] Jul 20 17:59:12.865995 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:12.865974 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:12.870165 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:12.870143 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 20 17:59:12.871052 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:12.871032 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8q44f\"" Jul 20 17:59:12.883310 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:12.883290 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75cmz"] Jul 20 17:59:13.007283 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.007251 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d032002b-b768-47e1-b32b-4c42377eede2-cert\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.007448 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.007303 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7jk\" (UniqueName: \"kubernetes.io/projected/d032002b-b768-47e1-b32b-4c42377eede2-kube-api-access-hz7jk\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.107934 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.107897 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d032002b-b768-47e1-b32b-4c42377eede2-cert\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.108119 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.107975 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7jk\" (UniqueName: \"kubernetes.io/projected/d032002b-b768-47e1-b32b-4c42377eede2-kube-api-access-hz7jk\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.110264 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.110211 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d032002b-b768-47e1-b32b-4c42377eede2-cert\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.118562 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.118541 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7jk\" (UniqueName: \"kubernetes.io/projected/d032002b-b768-47e1-b32b-4c42377eede2-kube-api-access-hz7jk\") pod \"odh-model-controller-7998c67dc4-75cmz\" (UID: \"d032002b-b768-47e1-b32b-4c42377eede2\") " pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.175467 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.175438 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:13.315517 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.315485 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-75cmz"] Jul 20 17:59:13.327374 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:13.327347 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd032002b_b768_47e1_b32b_4c42377eede2.slice/crio-0355b76aec82fd73da63f531af90f15b416eae724a9a269407fae8dda18c04d1 WatchSource:0}: Error finding container 0355b76aec82fd73da63f531af90f15b416eae724a9a269407fae8dda18c04d1: Status 404 returned error can't find the container with id 0355b76aec82fd73da63f531af90f15b416eae724a9a269407fae8dda18c04d1 Jul 20 17:59:13.813016 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.812980 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qp89n"] Jul 20 17:59:13.817362 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.817346 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:13.820090 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.820071 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 20 17:59:13.845528 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.845506 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qp89n"] Jul 20 17:59:13.915421 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.915390 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df491736-5bfe-4358-9538-7a3adb0fd84b-tls-certs\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:13.915421 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:13.915423 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flntt\" (UniqueName: \"kubernetes.io/projected/df491736-5bfe-4358-9538-7a3adb0fd84b-kube-api-access-flntt\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.016194 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.016106 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df491736-5bfe-4358-9538-7a3adb0fd84b-tls-certs\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.016194 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.016156 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flntt\" (UniqueName: \"kubernetes.io/projected/df491736-5bfe-4358-9538-7a3adb0fd84b-kube-api-access-flntt\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.019389 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.019354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df491736-5bfe-4358-9538-7a3adb0fd84b-tls-certs\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.033934 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.033906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flntt\" (UniqueName: \"kubernetes.io/projected/df491736-5bfe-4358-9538-7a3adb0fd84b-kube-api-access-flntt\") pod \"model-serving-api-54d89bd5fc-qp89n\" (UID: \"df491736-5bfe-4358-9538-7a3adb0fd84b\") " pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.126909 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.126828 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:14.237508 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.237443 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" event={"ID":"d032002b-b768-47e1-b32b-4c42377eede2","Type":"ContainerStarted","Data":"0355b76aec82fd73da63f531af90f15b416eae724a9a269407fae8dda18c04d1"} Jul 20 17:59:14.296182 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:14.296146 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qp89n"] Jul 20 17:59:14.300663 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:14.300614 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf491736_5bfe_4358_9538_7a3adb0fd84b.slice/crio-0299a0d4ec11ab9040ae48a30720fa49f540c8938ac5dc7b7bc18873951ee571 WatchSource:0}: Error finding container 0299a0d4ec11ab9040ae48a30720fa49f540c8938ac5dc7b7bc18873951ee571: Status 404 returned error can't find the container with id 0299a0d4ec11ab9040ae48a30720fa49f540c8938ac5dc7b7bc18873951ee571 Jul 20 17:59:15.242998 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:15.242959 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" event={"ID":"df491736-5bfe-4358-9538-7a3adb0fd84b","Type":"ContainerStarted","Data":"0299a0d4ec11ab9040ae48a30720fa49f540c8938ac5dc7b7bc18873951ee571"} Jul 20 17:59:16.248778 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:16.248751 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" event={"ID":"d032002b-b768-47e1-b32b-4c42377eede2","Type":"ContainerStarted","Data":"2406857bd75212eb88f1aeb9d4f8d1fd3b4014ce85e0cc48548184f402d1a8af"} Jul 20 17:59:16.249218 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:16.248885 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:16.274795 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:16.274740 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" podStartSLOduration=1.64961716 podStartE2EDuration="4.274724673s" totalImagesPullingTime="2.625107513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:16.272664028 +0000 UTC m=+684.210942337" watchObservedRunningTime="2026-07-20 17:59:16.274724673 +0000 UTC m=+684.213003007" Jul 20 17:59:17.814867 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.814783 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b99948476-qr9bh"] Jul 20 17:59:17.817902 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.817885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:17.821481 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.821460 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 17:59:17.821864 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.821847 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 17:59:17.821972 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.821934 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 17:59:17.822034 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.821962 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 17:59:17.822881 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.822864 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6cl9b\"" Jul 20 17:59:17.828893 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.828873 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b99948476-qr9bh"] Jul 20 17:59:17.946382 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.946351 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smspw\" (UniqueName: \"kubernetes.io/projected/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-kube-api-access-smspw\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:17.946548 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.946392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tmp\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:17.946548 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:17.946443 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.047026 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.046996 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.047187 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.047046 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smspw\" (UniqueName: \"kubernetes.io/projected/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-kube-api-access-smspw\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.047187 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.047070 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tmp\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.047187 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:59:18.047155 2554 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 20 17:59:18.047288 ip-10-0-135-108 kubenswrapper[2554]: E0720 17:59:18.047229 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs podName:8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c nodeName:}" failed. No retries permitted until 2026-07-20 17:59:18.547211573 +0000 UTC m=+686.485489841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs") pod "kube-auth-proxy-b99948476-qr9bh" (UID: "8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c") : secret "kube-auth-proxy-tls" not found Jul 20 17:59:18.049265 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.049237 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tmp\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.055976 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.055952 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smspw\" (UniqueName: \"kubernetes.io/projected/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-kube-api-access-smspw\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.257721 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.257688 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" event={"ID":"df491736-5bfe-4358-9538-7a3adb0fd84b","Type":"ContainerStarted","Data":"18ac269a6959606c6f0d0f0471deda279a8d41264f86006c7c510c0a13dc2978"} Jul 20 17:59:18.257875 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.257808 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:18.276453 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.276406 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" podStartSLOduration=2.065952999 podStartE2EDuration="5.27639182s" totalImagesPullingTime="3.210438821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:18.275242858 +0000 UTC m=+686.213521142" watchObservedRunningTime="2026-07-20 17:59:18.27639182 +0000 UTC m=+686.214670112" Jul 20 17:59:18.552095 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.552021 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.554291 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.554266 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c-tls-certs\") pod \"kube-auth-proxy-b99948476-qr9bh\" (UID: \"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c\") " pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.726502 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.726473 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" Jul 20 17:59:18.858507 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:18.858482 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b99948476-qr9bh"] Jul 20 17:59:18.861069 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:18.861043 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f73c6_c4a3_4a59_9e68_0e1d64b18f3c.slice/crio-7918045381cece4d56b5d0236f0119a872642d76d740644cd139e43256b8acb4 WatchSource:0}: Error finding container 7918045381cece4d56b5d0236f0119a872642d76d740644cd139e43256b8acb4: Status 404 returned error can't find the container with id 7918045381cece4d56b5d0236f0119a872642d76d740644cd139e43256b8acb4 Jul 20 17:59:19.262369 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.262334 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" event={"ID":"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c","Type":"ContainerStarted","Data":"7918045381cece4d56b5d0236f0119a872642d76d740644cd139e43256b8acb4"} Jul 20 17:59:19.652889 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.652803 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:19.656527 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.656492 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.659857 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.659812 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 17:59:19.660019 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.659934 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 17:59:19.660080 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.660055 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-888w8\"" Jul 20 17:59:19.660375 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.660354 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 17:59:19.660479 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.660361 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 17:59:19.681512 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.681484 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:19.761672 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.761672 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bc7\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.761928 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761711 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.761928 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.761928 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761857 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.761928 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761904 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.762089 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.761959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.862688 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862652 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862713 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862751 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862793 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bc7\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862824 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862858 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.863175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.862887 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.864068 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.863987 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.865792 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.865759 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.865887 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.865842 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.865998 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.865979 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.866172 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.866151 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.877457 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.877428 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.879776 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.879755 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bc7\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7\") pod \"istiod-openshift-gateway-7574459656-zr7c6\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:19.968065 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:19.967974 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:20.165852 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:20.165818 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:20.167531 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:20.167485 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03645f5f_854c_4431_ac85_0801481cdf6c.slice/crio-1463e26f6b7a0912e06991f6aab9cd3bffeca99b743765589d281c88991c31ce WatchSource:0}: Error finding container 1463e26f6b7a0912e06991f6aab9cd3bffeca99b743765589d281c88991c31ce: Status 404 returned error can't find the container with id 1463e26f6b7a0912e06991f6aab9cd3bffeca99b743765589d281c88991c31ce Jul 20 17:59:20.267111 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:20.267076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" event={"ID":"03645f5f-854c-4431-ac85-0801481cdf6c","Type":"ContainerStarted","Data":"1463e26f6b7a0912e06991f6aab9cd3bffeca99b743765589d281c88991c31ce"} Jul 20 17:59:23.057320 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.057285 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 17:59:23.057575 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.057350 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 17:59:23.283043 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.282946 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" event={"ID":"8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c","Type":"ContainerStarted","Data":"cbc88a3e13ebb37d5ffb8b0a92057ae48b155cf931f3928632c5aded85d59f41"} Jul 20 17:59:23.284511 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.284477 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" event={"ID":"03645f5f-854c-4431-ac85-0801481cdf6c","Type":"ContainerStarted","Data":"debae858720dbc968b66e53e9cce006e742b38e979905be742f9838d4a8251af"} Jul 20 17:59:23.284644 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.284533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:23.334739 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.334687 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b99948476-qr9bh" podStartSLOduration=2.192473843 podStartE2EDuration="6.33467238s" totalImagesPullingTime="4.142198537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:23.333722354 +0000 UTC m=+691.272000646" watchObservedRunningTime="2026-07-20 17:59:23.33467238 +0000 UTC m=+691.272950702" Jul 20 17:59:23.393685 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:23.393587 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" podStartSLOduration=1.506375268 podStartE2EDuration="4.393542272s" totalImagesPullingTime="2.887167004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:23.390860815 +0000 UTC m=+691.329139127" watchObservedRunningTime="2026-07-20 17:59:23.393542272 +0000 UTC m=+691.331820563" Jul 20 17:59:24.121967 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.121937 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cthmn"] Jul 20 17:59:24.125055 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.125036 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.128296 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.128273 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f5wv7\"" Jul 20 17:59:24.128443 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.128429 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 20 17:59:24.146238 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.146207 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cthmn"] Jul 20 17:59:24.147273 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.147252 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v"] Jul 20 17:59:24.150453 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.150430 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.153530 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.153501 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 20 17:59:24.154716 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.154697 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jtgtm\"" Jul 20 17:59:24.165487 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.165464 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v"] Jul 20 17:59:24.291462 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.291430 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zr7c6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 17:59:24.291621 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.291483 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 17:59:24.300790 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.300748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.300906 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.300834 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp6b\" (UniqueName: \"kubernetes.io/projected/818c0036-9703-485a-babb-631ceeb82bfb-kube-api-access-xjp6b\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.300906 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.300878 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818c0036-9703-485a-babb-631ceeb82bfb-cert\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.300973 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.300906 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7w7\" (UniqueName: \"kubernetes.io/projected/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-kube-api-access-kv7w7\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.401752 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.401660 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp6b\" (UniqueName: \"kubernetes.io/projected/818c0036-9703-485a-babb-631ceeb82bfb-kube-api-access-xjp6b\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.401752 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.401719 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818c0036-9703-485a-babb-631ceeb82bfb-cert\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.401985 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.401760 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7w7\" (UniqueName: \"kubernetes.io/projected/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-kube-api-access-kv7w7\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.401985 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.401947 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.404292 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.404267 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/818c0036-9703-485a-babb-631ceeb82bfb-cert\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.404447 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.404430 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.414425 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.414401 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7w7\" (UniqueName: \"kubernetes.io/projected/dd06f2ad-09ea-4eb4-adaf-cee1749697ec-kube-api-access-kv7w7\") pod \"llmisvc-controller-manager-777c6bd7d5-lh76v\" (UID: \"dd06f2ad-09ea-4eb4-adaf-cee1749697ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.416372 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.416355 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp6b\" (UniqueName: \"kubernetes.io/projected/818c0036-9703-485a-babb-631ceeb82bfb-kube-api-access-xjp6b\") pod \"kserve-controller-manager-d6f4b65b8-cthmn\" (UID: \"818c0036-9703-485a-babb-631ceeb82bfb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.435368 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.435344 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:24.460054 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.460033 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:24.595914 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.595879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cthmn"] Jul 20 17:59:24.602843 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:24.602813 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818c0036_9703_485a_babb_631ceeb82bfb.slice/crio-9d5403f49fa07595402446d3d1d5cf8d0bb9c47b7f0e8ba61758585fd68ec5e5 WatchSource:0}: Error finding container 9d5403f49fa07595402446d3d1d5cf8d0bb9c47b7f0e8ba61758585fd68ec5e5: Status 404 returned error can't find the container with id 9d5403f49fa07595402446d3d1d5cf8d0bb9c47b7f0e8ba61758585fd68ec5e5 Jul 20 17:59:24.640436 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:24.640400 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v"] Jul 20 17:59:24.644753 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:24.644727 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd06f2ad_09ea_4eb4_adaf_cee1749697ec.slice/crio-23031614f00a375a098ff824369e6efa01f805bab5a29ed42227faf297cf1f29 WatchSource:0}: Error finding container 23031614f00a375a098ff824369e6efa01f805bab5a29ed42227faf297cf1f29: Status 404 returned error can't find the container with id 23031614f00a375a098ff824369e6efa01f805bab5a29ed42227faf297cf1f29 Jul 20 17:59:25.285684 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.285650 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf"] Jul 20 17:59:25.288709 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.288685 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.292945 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.292923 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gc274\"" Jul 20 17:59:25.294595 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.294566 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" event={"ID":"dd06f2ad-09ea-4eb4-adaf-cee1749697ec","Type":"ContainerStarted","Data":"23031614f00a375a098ff824369e6efa01f805bab5a29ed42227faf297cf1f29"} Jul 20 17:59:25.296159 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.296031 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" event={"ID":"818c0036-9703-485a-babb-631ceeb82bfb","Type":"ContainerStarted","Data":"9d5403f49fa07595402446d3d1d5cf8d0bb9c47b7f0e8ba61758585fd68ec5e5"} Jul 20 17:59:25.325969 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.325941 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf"] Jul 20 17:59:25.411066 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411215 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411099 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411215 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411304 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411241 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411304 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411285 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85m5\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-kube-api-access-b85m5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411384 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411313 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411384 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411349 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411384 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411376 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.411521 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.411445 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.512853 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512807 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512939 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85m5\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-kube-api-access-b85m5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512963 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.512992 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513039 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513017 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513331 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513084 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513443 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513417 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513507 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513462 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513578 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513558 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.513641 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.513608 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.514045 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.514022 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.515307 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.515280 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.515573 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.515554 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.531787 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.531762 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85m5\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-kube-api-access-b85m5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.541530 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.541472 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45wngf\" (UID: \"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:25.599866 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:25.599835 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:26.137221 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:26.137185 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf"] Jul 20 17:59:26.140096 ip-10-0-135-108 kubenswrapper[2554]: W0720 17:59:26.140064 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38e833b_a5c0_4bd7_b2d9_9dc7d85107a9.slice/crio-8f13d885a16f4171ea9b9b63769fa2aac5cf77fd2a3f654cfe7c1dfbfeacb530 WatchSource:0}: Error finding container 8f13d885a16f4171ea9b9b63769fa2aac5cf77fd2a3f654cfe7c1dfbfeacb530: Status 404 returned error can't find the container with id 8f13d885a16f4171ea9b9b63769fa2aac5cf77fd2a3f654cfe7c1dfbfeacb530 Jul 20 17:59:26.302488 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:26.302449 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" event={"ID":"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9","Type":"ContainerStarted","Data":"8f13d885a16f4171ea9b9b63769fa2aac5cf77fd2a3f654cfe7c1dfbfeacb530"} Jul 20 17:59:27.255665 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:27.255604 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-75cmz" Jul 20 17:59:27.291264 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:27.291234 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:27.770413 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:27.770371 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:27.770899 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:27.770588 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" containerName="discovery" containerID="cri-o://debae858720dbc968b66e53e9cce006e742b38e979905be742f9838d4a8251af" gracePeriod=30 Jul 20 17:59:28.312962 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.312937 2554 generic.go:356] "Generic (PLEG): container finished" podID="03645f5f-854c-4431-ac85-0801481cdf6c" containerID="debae858720dbc968b66e53e9cce006e742b38e979905be742f9838d4a8251af" exitCode=0 Jul 20 17:59:28.313045 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.312993 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" event={"ID":"03645f5f-854c-4431-ac85-0801481cdf6c","Type":"ContainerDied","Data":"debae858720dbc968b66e53e9cce006e742b38e979905be742f9838d4a8251af"} Jul 20 17:59:28.425611 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.425559 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:28.541381 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541287 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541381 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541345 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541381 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541372 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541674 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541401 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541674 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541455 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541674 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541500 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7bc7\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.541674 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.541567 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert\") pod \"03645f5f-854c-4431-ac85-0801481cdf6c\" (UID: \"03645f5f-854c-4431-ac85-0801481cdf6c\") " Jul 20 17:59:28.542051 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.542015 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 17:59:28.544396 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544358 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:59:28.544396 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544369 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7" (OuterVolumeSpecName: "kube-api-access-f7bc7") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "kube-api-access-f7bc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:59:28.544396 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544358 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 17:59:28.544660 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544616 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs" (OuterVolumeSpecName: "local-certs") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 17:59:28.544660 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544656 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts" (OuterVolumeSpecName: "cacerts") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:59:28.544979 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.544955 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "03645f5f-854c-4431-ac85-0801481cdf6c" (UID: "03645f5f-854c-4431-ac85-0801481cdf6c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 17:59:28.643102 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643052 2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-cacerts\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643102 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643096 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bc7\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-kube-api-access-f7bc7\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643293 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643131 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-dns-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643293 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643147 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03645f5f-854c-4431-ac85-0801481cdf6c-istio-token\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643293 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643157 2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/03645f5f-854c-4431-ac85-0801481cdf6c-local-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643293 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643165 2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03645f5f-854c-4431-ac85-0801481cdf6c-istio-kubeconfig\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:28.643293 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:28.643178 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/03645f5f-854c-4431-ac85-0801481cdf6c-istio-csr-ca-configmap\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 17:59:29.267428 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.267377 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qp89n" Jul 20 17:59:29.320559 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.320446 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" event={"ID":"dd06f2ad-09ea-4eb4-adaf-cee1749697ec","Type":"ContainerStarted","Data":"b9af367fb866b02956f348e9f2c638ae72bb3a07c698e938e292dca2046f2aaf"} Jul 20 17:59:29.320559 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.320529 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 17:59:29.322463 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.322434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" event={"ID":"818c0036-9703-485a-babb-631ceeb82bfb","Type":"ContainerStarted","Data":"06b854ba9f4b5d4f155ad8bfbd06f0a5ccdc742899d31728ac03a69a0d004e2e"} Jul 20 17:59:29.322592 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.322490 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 17:59:29.324175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.324146 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" event={"ID":"03645f5f-854c-4431-ac85-0801481cdf6c","Type":"ContainerDied","Data":"1463e26f6b7a0912e06991f6aab9cd3bffeca99b743765589d281c88991c31ce"} Jul 20 17:59:29.324175 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.324161 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6" Jul 20 17:59:29.324367 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.324193 2554 scope.go:117] "RemoveContainer" containerID="debae858720dbc968b66e53e9cce006e742b38e979905be742f9838d4a8251af" Jul 20 17:59:29.339779 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.339733 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" podStartSLOduration=1.677929759 podStartE2EDuration="5.339718226s" totalImagesPullingTime="3.661788467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:29.339084041 +0000 UTC m=+697.277362332" watchObservedRunningTime="2026-07-20 17:59:29.339718226 +0000 UTC m=+697.277996516" Jul 20 17:59:29.355044 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.355012 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:29.359777 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.359747 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zr7c6"] Jul 20 17:59:29.374808 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:29.374764 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" podStartSLOduration=1.672913831 podStartE2EDuration="5.374747798s" totalImagesPullingTime="3.701833967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:29.373650425 +0000 UTC m=+697.311928728" watchObservedRunningTime="2026-07-20 17:59:29.374747798 +0000 UTC m=+697.313026087" Jul 20 17:59:30.395406 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:30.395371 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 17:59:30.395664 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:30.395445 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 17:59:30.395664 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:30.395483 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 17:59:30.550607 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:30.550563 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" path="/var/lib/kubelet/pods/03645f5f-854c-4431-ac85-0801481cdf6c/volumes" Jul 20 17:59:31.334916 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:31.334884 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" event={"ID":"e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9","Type":"ContainerStarted","Data":"5811964b4dc68d89ab8aba2e253da8d215d0b58d218e6e5b2ce1c2d5e8bb5067"} Jul 20 17:59:31.359164 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:31.359096 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" podStartSLOduration=2.106403683 podStartE2EDuration="6.359079924s" totalImagesPullingTime="4.252676241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:59:31.357189972 +0000 UTC m=+699.295468326" watchObservedRunningTime="2026-07-20 17:59:31.359079924 +0000 UTC m=+699.297358214" Jul 20 17:59:31.600334 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:31.600249 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:31.604851 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:31.604827 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:32.339539 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:32.339510 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 17:59:32.340576 ip-10-0-135-108 kubenswrapper[2554]: I0720 17:59:32.340558 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45wngf" Jul 20 18:00:00.331298 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:00.331268 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lh76v" Jul 20 18:00:00.334323 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:00.334303 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cthmn" Jul 20 18:00:13.672677 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.672639 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:13.673098 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.672915 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" containerName="discovery" Jul 20 18:00:13.673098 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.672925 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" containerName="discovery" Jul 20 18:00:13.673098 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.672974 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="03645f5f-854c-4431-ac85-0801481cdf6c" containerName="discovery" Jul 20 18:00:13.687922 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.687894 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:13.688127 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.687984 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:13.690680 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.690658 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 18:00:13.691833 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.691809 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 18:00:13.691929 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.691892 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sh7kz\"" Jul 20 18:00:13.754956 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.754931 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dzr\" (UniqueName: \"kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr\") pod \"kuadrant-operator-catalog-5pjnj\" (UID: \"053d4f92-b39e-498e-99d5-e8171565a85b\") " pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:13.856003 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.855977 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dzr\" (UniqueName: \"kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr\") pod \"kuadrant-operator-catalog-5pjnj\" (UID: \"053d4f92-b39e-498e-99d5-e8171565a85b\") " pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:13.865160 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:13.865137 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dzr\" (UniqueName: \"kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr\") pod \"kuadrant-operator-catalog-5pjnj\" (UID: \"053d4f92-b39e-498e-99d5-e8171565a85b\") " pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:14.000184 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.000156 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:14.034666 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.034640 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:14.134875 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.134786 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:14.137559 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:14.137524 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053d4f92_b39e_498e_99d5_e8171565a85b.slice/crio-9dddb4d91457f7a1b25088e0251d7368f3f150a5ab7d984ee36dc57e4217fc85 WatchSource:0}: Error finding container 9dddb4d91457f7a1b25088e0251d7368f3f150a5ab7d984ee36dc57e4217fc85: Status 404 returned error can't find the container with id 9dddb4d91457f7a1b25088e0251d7368f3f150a5ab7d984ee36dc57e4217fc85 Jul 20 18:00:14.246682 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.246658 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4vs7j"] Jul 20 18:00:14.250959 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.250919 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:14.261570 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.261541 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4vs7j"] Jul 20 18:00:14.359768 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.359745 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvl7\" (UniqueName: \"kubernetes.io/projected/79e7c853-db8a-4810-a9a8-87d588931153-kube-api-access-zrvl7\") pod \"kuadrant-operator-catalog-4vs7j\" (UID: \"79e7c853-db8a-4810-a9a8-87d588931153\") " pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:14.460769 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.460737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvl7\" (UniqueName: \"kubernetes.io/projected/79e7c853-db8a-4810-a9a8-87d588931153-kube-api-access-zrvl7\") pod \"kuadrant-operator-catalog-4vs7j\" (UID: \"79e7c853-db8a-4810-a9a8-87d588931153\") " pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:14.469342 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.469321 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvl7\" (UniqueName: \"kubernetes.io/projected/79e7c853-db8a-4810-a9a8-87d588931153-kube-api-access-zrvl7\") pod \"kuadrant-operator-catalog-4vs7j\" (UID: \"79e7c853-db8a-4810-a9a8-87d588931153\") " pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:14.508410 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.508356 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" event={"ID":"053d4f92-b39e-498e-99d5-e8171565a85b","Type":"ContainerStarted","Data":"9dddb4d91457f7a1b25088e0251d7368f3f150a5ab7d984ee36dc57e4217fc85"} Jul 20 18:00:14.559712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.559691 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:14.685718 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:14.685690 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4vs7j"] Jul 20 18:00:14.687947 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:14.687922 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e7c853_db8a_4810_a9a8_87d588931153.slice/crio-11acd192631241f7654f591781d5c20f7fa73b47b01cf1db62c3d36e762681d4 WatchSource:0}: Error finding container 11acd192631241f7654f591781d5c20f7fa73b47b01cf1db62c3d36e762681d4: Status 404 returned error can't find the container with id 11acd192631241f7654f591781d5c20f7fa73b47b01cf1db62c3d36e762681d4 Jul 20 18:00:15.517391 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:15.517354 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" event={"ID":"79e7c853-db8a-4810-a9a8-87d588931153","Type":"ContainerStarted","Data":"11acd192631241f7654f591781d5c20f7fa73b47b01cf1db62c3d36e762681d4"} Jul 20 18:00:18.533548 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.533500 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" event={"ID":"053d4f92-b39e-498e-99d5-e8171565a85b","Type":"ContainerStarted","Data":"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11"} Jul 20 18:00:18.534025 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.533595 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" podUID="053d4f92-b39e-498e-99d5-e8171565a85b" containerName="registry-server" containerID="cri-o://f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11" gracePeriod=2 Jul 20 18:00:18.535000 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.534976 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" event={"ID":"79e7c853-db8a-4810-a9a8-87d588931153","Type":"ContainerStarted","Data":"14f0c3cc2d5f4d92bbf2848a987da19110ad8fa81a2a77721a1e904700856729"} Jul 20 18:00:18.550721 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.550682 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" podStartSLOduration=1.83289871 podStartE2EDuration="5.550669102s" totalImagesPullingTime="3.717770392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:18.549269087 +0000 UTC m=+746.487547408" watchObservedRunningTime="2026-07-20 18:00:18.550669102 +0000 UTC m=+746.488947391" Jul 20 18:00:18.567168 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.567125 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" podStartSLOduration=1.396228508 podStartE2EDuration="4.567112868s" totalImagesPullingTime="3.17088436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:18.565787651 +0000 UTC m=+746.504065952" watchObservedRunningTime="2026-07-20 18:00:18.567112868 +0000 UTC m=+746.505391156" Jul 20 18:00:18.761912 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.761882 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:18.796809 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.796740 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6dzr\" (UniqueName: \"kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr\") pod \"053d4f92-b39e-498e-99d5-e8171565a85b\" (UID: \"053d4f92-b39e-498e-99d5-e8171565a85b\") " Jul 20 18:00:18.798831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.798805 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr" (OuterVolumeSpecName: "kube-api-access-j6dzr") pod "053d4f92-b39e-498e-99d5-e8171565a85b" (UID: "053d4f92-b39e-498e-99d5-e8171565a85b"). InnerVolumeSpecName "kube-api-access-j6dzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:00:18.898058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:18.898036 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dzr\" (UniqueName: \"kubernetes.io/projected/053d4f92-b39e-498e-99d5-e8171565a85b-kube-api-access-j6dzr\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:00:19.540643 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.540596 2554 generic.go:356] "Generic (PLEG): container finished" podID="053d4f92-b39e-498e-99d5-e8171565a85b" containerID="f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11" exitCode=0 Jul 20 18:00:19.541058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.540657 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" Jul 20 18:00:19.541058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.540680 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" event={"ID":"053d4f92-b39e-498e-99d5-e8171565a85b","Type":"ContainerDied","Data":"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11"} Jul 20 18:00:19.541058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.540725 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pjnj" event={"ID":"053d4f92-b39e-498e-99d5-e8171565a85b","Type":"ContainerDied","Data":"9dddb4d91457f7a1b25088e0251d7368f3f150a5ab7d984ee36dc57e4217fc85"} Jul 20 18:00:19.541058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.540745 2554 scope.go:117] "RemoveContainer" containerID="f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11" Jul 20 18:00:19.549481 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.549467 2554 scope.go:117] "RemoveContainer" containerID="f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11" Jul 20 18:00:19.549777 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:00:19.549744 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11\": container with ID starting with f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11 not found: ID does not exist" containerID="f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11" Jul 20 18:00:19.549819 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.549788 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11"} err="failed to get container status \"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11\": rpc error: code = NotFound desc = could not find container \"f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11\": container with ID starting with f7a03e2b2d6e2113cda0c3871580fda32647feb5323fbde30ef2a56274e0ec11 not found: ID does not exist" Jul 20 18:00:19.562263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.562239 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:19.566144 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:19.566126 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pjnj"] Jul 20 18:00:20.548644 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:20.548600 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053d4f92-b39e-498e-99d5-e8171565a85b" path="/var/lib/kubelet/pods/053d4f92-b39e-498e-99d5-e8171565a85b/volumes" Jul 20 18:00:24.559811 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:24.559777 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:24.560310 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:24.559830 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:24.581979 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:24.581955 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:25.586232 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:25.586203 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4vs7j" Jul 20 18:00:47.852600 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.852565 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw"] Jul 20 18:00:47.853061 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.852865 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053d4f92-b39e-498e-99d5-e8171565a85b" containerName="registry-server" Jul 20 18:00:47.853061 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.852877 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="053d4f92-b39e-498e-99d5-e8171565a85b" containerName="registry-server" Jul 20 18:00:47.853061 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.852945 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="053d4f92-b39e-498e-99d5-e8171565a85b" containerName="registry-server" Jul 20 18:00:47.855694 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.855674 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:47.858324 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.858305 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jftbm\"" Jul 20 18:00:47.870828 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:47.870807 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw"] Jul 20 18:00:48.007672 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.007622 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg56m\" (UniqueName: \"kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.007848 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.007689 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.108216 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.108128 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.108216 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.108208 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg56m\" (UniqueName: \"kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.108476 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.108458 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.118872 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.118850 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg56m\" (UniqueName: \"kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.164857 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.164825 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:48.292755 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.292723 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw"] Jul 20 18:00:48.295932 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:48.295899 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d67676c_3443_4f25_936a_7f682a76a2e0.slice/crio-18e0b35b1d0dde5fb6741e23e11b0e5cc2421b9e5d94c23f2ac431b43f137a40 WatchSource:0}: Error finding container 18e0b35b1d0dde5fb6741e23e11b0e5cc2421b9e5d94c23f2ac431b43f137a40: Status 404 returned error can't find the container with id 18e0b35b1d0dde5fb6741e23e11b0e5cc2421b9e5d94c23f2ac431b43f137a40 Jul 20 18:00:48.649383 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:48.649348 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" event={"ID":"0d67676c-3443-4f25-936a-7f682a76a2e0","Type":"ContainerStarted","Data":"18e0b35b1d0dde5fb6741e23e11b0e5cc2421b9e5d94c23f2ac431b43f137a40"} Jul 20 18:00:49.799472 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:49.799425 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s"] Jul 20 18:00:49.803340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:49.803313 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:49.806138 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:49.806114 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p2n49\"" Jul 20 18:00:49.812962 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:49.812934 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s"] Jul 20 18:00:49.921835 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:49.921799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2dz\" (UniqueName: \"kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz\") pod \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" (UID: \"cbb62c7c-be36-43b8-8000-ce69183d3cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:50.023121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:50.023086 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2dz\" (UniqueName: \"kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz\") pod \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" (UID: \"cbb62c7c-be36-43b8-8000-ce69183d3cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:50.031853 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:50.031814 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2dz\" (UniqueName: \"kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz\") pod \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" (UID: \"cbb62c7c-be36-43b8-8000-ce69183d3cdf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:50.115664 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:50.115555 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:50.249366 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:50.249318 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s"] Jul 20 18:00:50.252265 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:50.252226 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb62c7c_be36_43b8_8000_ce69183d3cdf.slice/crio-5e5ed749d62270cd380e53bf4f5f67a1a5d39f2a1b996f2c4d99c6f748d20b34 WatchSource:0}: Error finding container 5e5ed749d62270cd380e53bf4f5f67a1a5d39f2a1b996f2c4d99c6f748d20b34: Status 404 returned error can't find the container with id 5e5ed749d62270cd380e53bf4f5f67a1a5d39f2a1b996f2c4d99c6f748d20b34 Jul 20 18:00:50.660393 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:50.660360 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" event={"ID":"cbb62c7c-be36-43b8-8000-ce69183d3cdf","Type":"ContainerStarted","Data":"5e5ed749d62270cd380e53bf4f5f67a1a5d39f2a1b996f2c4d99c6f748d20b34"} Jul 20 18:00:52.459325 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.459294 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562"] Jul 20 18:00:52.462554 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.462538 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:52.465336 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.465310 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l75w4\"" Jul 20 18:00:52.465519 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.465503 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 20 18:00:52.477326 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.477302 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562"] Jul 20 18:00:52.547357 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.547316 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2jt\" (UniqueName: \"kubernetes.io/projected/f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680-kube-api-access-zm2jt\") pod \"dns-operator-controller-manager-864dd78c4c-vq562\" (UID: \"f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:52.648109 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.648078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2jt\" (UniqueName: \"kubernetes.io/projected/f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680-kube-api-access-zm2jt\") pod \"dns-operator-controller-manager-864dd78c4c-vq562\" (UID: \"f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:52.660876 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.660851 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2jt\" (UniqueName: \"kubernetes.io/projected/f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680-kube-api-access-zm2jt\") pod \"dns-operator-controller-manager-864dd78c4c-vq562\" (UID: \"f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:52.773225 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.773199 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:52.920052 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:52.920024 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562"] Jul 20 18:00:52.922710 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:52.922679 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a9c81e_d746_4fd3_9f19_4dbe5ae8c680.slice/crio-ef50df4f025fb77941ab39357eafc8be30e8d5c3d000d081d77f27d68f9a6e0d WatchSource:0}: Error finding container ef50df4f025fb77941ab39357eafc8be30e8d5c3d000d081d77f27d68f9a6e0d: Status 404 returned error can't find the container with id ef50df4f025fb77941ab39357eafc8be30e8d5c3d000d081d77f27d68f9a6e0d Jul 20 18:00:53.677577 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:53.677534 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" event={"ID":"f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680","Type":"ContainerStarted","Data":"ef50df4f025fb77941ab39357eafc8be30e8d5c3d000d081d77f27d68f9a6e0d"} Jul 20 18:00:56.099768 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.099735 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg"] Jul 20 18:00:56.103341 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.103322 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:56.106131 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.106113 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6q8kz\"" Jul 20 18:00:56.115874 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.115814 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg"] Jul 20 18:00:56.177934 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.177902 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxchp\" (UniqueName: \"kubernetes.io/projected/b40d9a41-4fcc-48bb-ae51-6acad5a9034d-kube-api-access-vxchp\") pod \"authorino-operator-75cc8d5cb7-z2ptg\" (UID: \"b40d9a41-4fcc-48bb-ae51-6acad5a9034d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:56.278610 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.278577 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxchp\" (UniqueName: \"kubernetes.io/projected/b40d9a41-4fcc-48bb-ae51-6acad5a9034d-kube-api-access-vxchp\") pod \"authorino-operator-75cc8d5cb7-z2ptg\" (UID: \"b40d9a41-4fcc-48bb-ae51-6acad5a9034d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:56.288261 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.288239 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxchp\" (UniqueName: \"kubernetes.io/projected/b40d9a41-4fcc-48bb-ae51-6acad5a9034d-kube-api-access-vxchp\") pod \"authorino-operator-75cc8d5cb7-z2ptg\" (UID: \"b40d9a41-4fcc-48bb-ae51-6acad5a9034d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:56.414805 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.414703 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:56.696733 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.696611 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" event={"ID":"cbb62c7c-be36-43b8-8000-ce69183d3cdf","Type":"ContainerStarted","Data":"d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580"} Jul 20 18:00:56.696733 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.696696 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:00:56.698193 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.698173 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" event={"ID":"0d67676c-3443-4f25-936a-7f682a76a2e0","Type":"ContainerStarted","Data":"d3c098bfd948a70e6c0ba405181bfe6ef160ee53f5cfb332e846cb6a3686d579"} Jul 20 18:00:56.698324 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.698308 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:00:56.716914 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.716665 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" podStartSLOduration=1.762190511 podStartE2EDuration="7.716648868s" totalImagesPullingTime="5.954458357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:56.716492575 +0000 UTC m=+784.654770890" watchObservedRunningTime="2026-07-20 18:00:56.716648868 +0000 UTC m=+784.654927155" Jul 20 18:00:56.743132 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.743091 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg"] Jul 20 18:00:56.744045 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:56.744002 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" podStartSLOduration=1.998569157 podStartE2EDuration="9.743988974s" totalImagesPullingTime="7.745419817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:56.740911204 +0000 UTC m=+784.679189512" watchObservedRunningTime="2026-07-20 18:00:56.743988974 +0000 UTC m=+784.682267263" Jul 20 18:00:56.745053 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:00:56.745031 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40d9a41_4fcc_48bb_ae51_6acad5a9034d.slice/crio-6cf6927cfb984590980ce172a3d055514ca4aaa27954bef26807eaa6ac5199df WatchSource:0}: Error finding container 6cf6927cfb984590980ce172a3d055514ca4aaa27954bef26807eaa6ac5199df: Status 404 returned error can't find the container with id 6cf6927cfb984590980ce172a3d055514ca4aaa27954bef26807eaa6ac5199df Jul 20 18:00:57.705094 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:57.705053 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" event={"ID":"b40d9a41-4fcc-48bb-ae51-6acad5a9034d","Type":"ContainerStarted","Data":"6cf6927cfb984590980ce172a3d055514ca4aaa27954bef26807eaa6ac5199df"} Jul 20 18:00:57.706917 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:57.706888 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" event={"ID":"f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680","Type":"ContainerStarted","Data":"6c6c657e258f187de582cbceb32b4587b8d5cd164ec475e8d7d1c0840c2297de"} Jul 20 18:00:57.725574 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:57.725338 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" podStartSLOduration=2.002958816 podStartE2EDuration="5.725322428s" totalImagesPullingTime="3.722363612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:57.724248735 +0000 UTC m=+785.662527061" watchObservedRunningTime="2026-07-20 18:00:57.725322428 +0000 UTC m=+785.663600720" Jul 20 18:00:58.711999 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:58.711965 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:00:59.717508 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:59.717464 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" event={"ID":"b40d9a41-4fcc-48bb-ae51-6acad5a9034d","Type":"ContainerStarted","Data":"d97cc42c37cf3acd8e21a1cd012d868a40c205bede925be2260170632d93f51d"} Jul 20 18:00:59.717976 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:59.717645 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:00:59.735920 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:00:59.735872 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" podStartSLOduration=1.589484266 podStartE2EDuration="3.735856856s" totalImagesPullingTime="2.14637259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:59.734828928 +0000 UTC m=+787.673107244" watchObservedRunningTime="2026-07-20 18:00:59.735856856 +0000 UTC m=+787.674135185" Jul 20 18:01:07.709425 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:07.709394 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:07.709894 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:07.709561 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.635198 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.635165 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw"] Jul 20 18:01:09.635605 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.635356 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" containerName="manager" containerID="cri-o://d3c098bfd948a70e6c0ba405181bfe6ef160ee53f5cfb332e846cb6a3686d579" gracePeriod=2 Jul 20 18:01:09.641427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.641399 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw"] Jul 20 18:01:09.655265 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.655238 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s"] Jul 20 18:01:09.655484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.655460 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" containerName="manager" containerID="cri-o://d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580" gracePeriod=2 Jul 20 18:01:09.659942 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.659908 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.665581 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.665560 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:09.666271 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.666243 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" containerName="manager" Jul 20 18:01:09.666271 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.666273 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" containerName="manager" Jul 20 18:01:09.666402 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.666371 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" containerName="manager" Jul 20 18:01:09.669410 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.669390 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s"] Jul 20 18:01:09.669528 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.669473 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.683466 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.683442 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf"] Jul 20 18:01:09.683780 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.683765 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" containerName="manager" Jul 20 18:01:09.683780 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.683781 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" containerName="manager" Jul 20 18:01:09.683904 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.683833 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" containerName="manager" Jul 20 18:01:09.686894 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.686851 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:09.687015 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.686977 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:09.688749 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.688723 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.688852 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.688773 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m49p\" (UniqueName: \"kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.696292 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.696270 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf"] Jul 20 18:01:09.707333 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.707305 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.709555 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.709528 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.711659 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.711620 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.719863 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.719844 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq562" Jul 20 18:01:09.741781 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.741750 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.761137 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.761016 2554 generic.go:356] "Generic (PLEG): container finished" podID="0d67676c-3443-4f25-936a-7f682a76a2e0" containerID="d3c098bfd948a70e6c0ba405181bfe6ef160ee53f5cfb332e846cb6a3686d579" exitCode=0 Jul 20 18:01:09.762110 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.762074 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.764126 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.764100 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.790296 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.790248 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvmj\" (UniqueName: \"kubernetes.io/projected/dab3ab50-e988-4b19-90fb-2506ed0768f8-kube-api-access-llvmj\") pod \"limitador-operator-controller-manager-859d7b78bd-pn7sf\" (UID: \"dab3ab50-e988-4b19-90fb-2506ed0768f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:09.790426 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.790354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.790466 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.790424 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m49p\" (UniqueName: \"kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.790864 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.790840 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.800346 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.800322 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m49p\" (UniqueName: \"kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mptb4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:09.891749 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.891657 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvmj\" (UniqueName: \"kubernetes.io/projected/dab3ab50-e988-4b19-90fb-2506ed0768f8-kube-api-access-llvmj\") pod \"limitador-operator-controller-manager-859d7b78bd-pn7sf\" (UID: \"dab3ab50-e988-4b19-90fb-2506ed0768f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:09.900690 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.900581 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvmj\" (UniqueName: \"kubernetes.io/projected/dab3ab50-e988-4b19-90fb-2506ed0768f8-kube-api-access-llvmj\") pod \"limitador-operator-controller-manager-859d7b78bd-pn7sf\" (UID: \"dab3ab50-e988-4b19-90fb-2506ed0768f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:09.918770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.918748 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.920834 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.920815 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.921524 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.921504 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.923989 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.923966 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.926124 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.926096 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tqj8s\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:09.928296 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:09.928270 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mdbzw\" is forbidden: User \"system:node:ip-10-0-135-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-108.ec2.internal' and this object" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:09.992264 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.992234 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f2dz\" (UniqueName: \"kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz\") pod \"cbb62c7c-be36-43b8-8000-ce69183d3cdf\" (UID: \"cbb62c7c-be36-43b8-8000-ce69183d3cdf\") " Jul 20 18:01:09.992418 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.992275 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg56m\" (UniqueName: \"kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m\") pod \"0d67676c-3443-4f25-936a-7f682a76a2e0\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " Jul 20 18:01:09.992418 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.992300 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume\") pod \"0d67676c-3443-4f25-936a-7f682a76a2e0\" (UID: \"0d67676c-3443-4f25-936a-7f682a76a2e0\") " Jul 20 18:01:09.992849 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.992811 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0d67676c-3443-4f25-936a-7f682a76a2e0" (UID: "0d67676c-3443-4f25-936a-7f682a76a2e0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:01:09.994344 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.994319 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m" (OuterVolumeSpecName: "kube-api-access-pg56m") pod "0d67676c-3443-4f25-936a-7f682a76a2e0" (UID: "0d67676c-3443-4f25-936a-7f682a76a2e0"). InnerVolumeSpecName "kube-api-access-pg56m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:01:09.994430 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:09.994360 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz" (OuterVolumeSpecName: "kube-api-access-4f2dz") pod "cbb62c7c-be36-43b8-8000-ce69183d3cdf" (UID: "cbb62c7c-be36-43b8-8000-ce69183d3cdf"). InnerVolumeSpecName "kube-api-access-4f2dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:01:10.076597 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.076565 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:10.084579 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.084557 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:10.093939 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.093913 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg56m\" (UniqueName: \"kubernetes.io/projected/0d67676c-3443-4f25-936a-7f682a76a2e0-kube-api-access-pg56m\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:01:10.094037 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.093943 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d67676c-3443-4f25-936a-7f682a76a2e0-extensions-socket-volume\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:01:10.094037 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.093958 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2dz\" (UniqueName: \"kubernetes.io/projected/cbb62c7c-be36-43b8-8000-ce69183d3cdf-kube-api-access-4f2dz\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:01:10.239847 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.239815 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:10.243371 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:01:10.243346 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952a561f_ed0f_4963_886d_0bde203e89b4.slice/crio-74f83251a627f3e3f648adc5b61664e3f1e91bdc25cd7944e0c6df52515aa307 WatchSource:0}: Error finding container 74f83251a627f3e3f648adc5b61664e3f1e91bdc25cd7944e0c6df52515aa307: Status 404 returned error can't find the container with id 74f83251a627f3e3f648adc5b61664e3f1e91bdc25cd7944e0c6df52515aa307 Jul 20 18:01:10.252479 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.252456 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf"] Jul 20 18:01:10.253518 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:01:10.253496 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab3ab50_e988_4b19_90fb_2506ed0768f8.slice/crio-1bbd914a7e3694010ab2d5dc34ef8b38aba1aa2c1c278e9d6f6913361e1628c7 WatchSource:0}: Error finding container 1bbd914a7e3694010ab2d5dc34ef8b38aba1aa2c1c278e9d6f6913361e1628c7: Status 404 returned error can't find the container with id 1bbd914a7e3694010ab2d5dc34ef8b38aba1aa2c1c278e9d6f6913361e1628c7 Jul 20 18:01:10.549752 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.549721 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d67676c-3443-4f25-936a-7f682a76a2e0" path="/var/lib/kubelet/pods/0d67676c-3443-4f25-936a-7f682a76a2e0/volumes" Jul 20 18:01:10.550045 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.550033 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" path="/var/lib/kubelet/pods/cbb62c7c-be36-43b8-8000-ce69183d3cdf/volumes" Jul 20 18:01:10.724121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.724087 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-z2ptg" Jul 20 18:01:10.766383 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.766353 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mdbzw" Jul 20 18:01:10.766383 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.766363 2554 scope.go:117] "RemoveContainer" containerID="d3c098bfd948a70e6c0ba405181bfe6ef160ee53f5cfb332e846cb6a3686d579" Jul 20 18:01:10.768161 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.768130 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" event={"ID":"952a561f-ed0f-4963-886d-0bde203e89b4","Type":"ContainerStarted","Data":"75f8f0883c497e75c660a7b17f1a85967c7101443c4894e44f9dbed1a399b803"} Jul 20 18:01:10.768281 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.768257 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:10.768397 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.768309 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" event={"ID":"952a561f-ed0f-4963-886d-0bde203e89b4","Type":"ContainerStarted","Data":"74f83251a627f3e3f648adc5b61664e3f1e91bdc25cd7944e0c6df52515aa307"} Jul 20 18:01:10.770027 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.769990 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" event={"ID":"dab3ab50-e988-4b19-90fb-2506ed0768f8","Type":"ContainerStarted","Data":"1f5da2712872d6ea56a7682bf3bb5c98e01f6f37f00a41c89bcf4ab2171d0025"} Jul 20 18:01:10.770180 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.770072 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" event={"ID":"dab3ab50-e988-4b19-90fb-2506ed0768f8","Type":"ContainerStarted","Data":"1bbd914a7e3694010ab2d5dc34ef8b38aba1aa2c1c278e9d6f6913361e1628c7"} Jul 20 18:01:10.770180 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.770104 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:10.771604 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.771582 2554 generic.go:356] "Generic (PLEG): container finished" podID="cbb62c7c-be36-43b8-8000-ce69183d3cdf" containerID="d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580" exitCode=0 Jul 20 18:01:10.771710 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.771683 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tqj8s" Jul 20 18:01:10.776188 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.776172 2554 scope.go:117] "RemoveContainer" containerID="d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580" Jul 20 18:01:10.785119 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.785103 2554 scope.go:117] "RemoveContainer" containerID="d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580" Jul 20 18:01:10.785417 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:01:10.785391 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580\": container with ID starting with d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580 not found: ID does not exist" containerID="d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580" Jul 20 18:01:10.785463 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.785429 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580"} err="failed to get container status \"d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580\": rpc error: code = NotFound desc = could not find container \"d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580\": container with ID starting with d283d5841b7b85b8d0e3be320a62f4439b349616fb5cdcaea09668963babf580 not found: ID does not exist" Jul 20 18:01:10.800796 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.800723 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" podStartSLOduration=1.800709365 podStartE2EDuration="1.800709365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:01:10.799956385 +0000 UTC m=+798.738234972" watchObservedRunningTime="2026-07-20 18:01:10.800709365 +0000 UTC m=+798.738987652" Jul 20 18:01:10.823295 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:10.823252 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" podStartSLOduration=1.823239348 podStartE2EDuration="1.823239348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:01:10.822718139 +0000 UTC m=+798.760996430" watchObservedRunningTime="2026-07-20 18:01:10.823239348 +0000 UTC m=+798.761517637" Jul 20 18:01:21.780048 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:21.780011 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pn7sf" Jul 20 18:01:21.780659 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:21.780093 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:44.783726 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:44.783689 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:44.784192 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:44.783979 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" podUID="952a561f-ed0f-4963-886d-0bde203e89b4" containerName="manager" containerID="cri-o://75f8f0883c497e75c660a7b17f1a85967c7101443c4894e44f9dbed1a399b803" gracePeriod=10 Jul 20 18:01:44.920433 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:44.920404 2554 generic.go:356] "Generic (PLEG): container finished" podID="952a561f-ed0f-4963-886d-0bde203e89b4" containerID="75f8f0883c497e75c660a7b17f1a85967c7101443c4894e44f9dbed1a399b803" exitCode=0 Jul 20 18:01:44.920610 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:44.920473 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" event={"ID":"952a561f-ed0f-4963-886d-0bde203e89b4","Type":"ContainerDied","Data":"75f8f0883c497e75c660a7b17f1a85967c7101443c4894e44f9dbed1a399b803"} Jul 20 18:01:45.035081 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.035024 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:45.182341 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.182306 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m49p\" (UniqueName: \"kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p\") pod \"952a561f-ed0f-4963-886d-0bde203e89b4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " Jul 20 18:01:45.182514 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.182363 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume\") pod \"952a561f-ed0f-4963-886d-0bde203e89b4\" (UID: \"952a561f-ed0f-4963-886d-0bde203e89b4\") " Jul 20 18:01:45.182846 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.182821 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "952a561f-ed0f-4963-886d-0bde203e89b4" (UID: "952a561f-ed0f-4963-886d-0bde203e89b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:01:45.184488 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.184466 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p" (OuterVolumeSpecName: "kube-api-access-8m49p") pod "952a561f-ed0f-4963-886d-0bde203e89b4" (UID: "952a561f-ed0f-4963-886d-0bde203e89b4"). InnerVolumeSpecName "kube-api-access-8m49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:01:45.283811 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.283769 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952a561f-ed0f-4963-886d-0bde203e89b4-extensions-socket-volume\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:01:45.283811 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.283802 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m49p\" (UniqueName: \"kubernetes.io/projected/952a561f-ed0f-4963-886d-0bde203e89b4-kube-api-access-8m49p\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:01:45.926072 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.926040 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" Jul 20 18:01:45.926502 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.926072 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4" event={"ID":"952a561f-ed0f-4963-886d-0bde203e89b4","Type":"ContainerDied","Data":"74f83251a627f3e3f648adc5b61664e3f1e91bdc25cd7944e0c6df52515aa307"} Jul 20 18:01:45.926502 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.926130 2554 scope.go:117] "RemoveContainer" containerID="75f8f0883c497e75c660a7b17f1a85967c7101443c4894e44f9dbed1a399b803" Jul 20 18:01:45.951534 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.951511 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:45.958613 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:45.958585 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mptb4"] Jul 20 18:01:46.549176 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:01:46.549147 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952a561f-ed0f-4963-886d-0bde203e89b4" path="/var/lib/kubelet/pods/952a561f-ed0f-4963-886d-0bde203e89b4/volumes" Jul 20 18:02:01.223947 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.223917 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp"] Jul 20 18:02:01.224378 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.224228 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952a561f-ed0f-4963-886d-0bde203e89b4" containerName="manager" Jul 20 18:02:01.224378 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.224239 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="952a561f-ed0f-4963-886d-0bde203e89b4" containerName="manager" Jul 20 18:02:01.224378 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.224307 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="952a561f-ed0f-4963-886d-0bde203e89b4" containerName="manager" Jul 20 18:02:01.228561 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.228542 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.236538 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.236519 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6fmpl\"" Jul 20 18:02:01.255291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.255265 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp"] Jul 20 18:02:01.408845 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.408813 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.408852 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee786b93-3864-4931-8670-b7b1736db018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.408883 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbrl\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-kube-api-access-2bbrl\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.408971 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.409012 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee786b93-3864-4931-8670-b7b1736db018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.409035 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.409102 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.409138 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.409197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.409166 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510482 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee786b93-3864-4931-8670-b7b1736db018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510519 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510540 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510561 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510598 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510616 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee786b93-3864-4931-8670-b7b1736db018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.510686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510658 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbrl\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-kube-api-access-2bbrl\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511034 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510694 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511034 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510928 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511034 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.510969 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511179 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.511120 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511179 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.511137 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.511382 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.511364 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee786b93-3864-4931-8670-b7b1736db018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.512976 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.512954 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee786b93-3864-4931-8670-b7b1736db018-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.513172 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.513153 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee786b93-3864-4931-8670-b7b1736db018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.519563 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.519541 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.519747 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.519728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbrl\" (UniqueName: \"kubernetes.io/projected/ee786b93-3864-4931-8670-b7b1736db018-kube-api-access-2bbrl\") pod \"maas-default-gateway-openshift-default-56654dcf7-tlcgp\" (UID: \"ee786b93-3864-4931-8670-b7b1736db018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.538401 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.538377 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:01.672724 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.672699 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp"] Jul 20 18:02:01.675171 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:01.675141 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee786b93_3864_4931_8670_b7b1736db018.slice/crio-766e70d4bb3c68fd7bdd50211d6a97574887017e5b90d2a314ae2f3cc10c4b36 WatchSource:0}: Error finding container 766e70d4bb3c68fd7bdd50211d6a97574887017e5b90d2a314ae2f3cc10c4b36: Status 404 returned error can't find the container with id 766e70d4bb3c68fd7bdd50211d6a97574887017e5b90d2a314ae2f3cc10c4b36 Jul 20 18:02:01.682594 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.682556 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:02:01.682905 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.682875 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:02:01.683145 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.683123 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:02:01.997551 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.997455 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" event={"ID":"ee786b93-3864-4931-8670-b7b1736db018","Type":"ContainerStarted","Data":"1442851e1463bbf5cb2aed5b90f57dffe46a1c4266e6a91f470259ffc235e754"} Jul 20 18:02:01.997551 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:01.997539 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" event={"ID":"ee786b93-3864-4931-8670-b7b1736db018","Type":"ContainerStarted","Data":"766e70d4bb3c68fd7bdd50211d6a97574887017e5b90d2a314ae2f3cc10c4b36"} Jul 20 18:02:02.539149 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:02.539116 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:02.549796 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:02.549773 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:02.573283 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:02.573240 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" podStartSLOduration=1.57322645 podStartE2EDuration="1.57322645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:02.019461808 +0000 UTC m=+849.957740099" watchObservedRunningTime="2026-07-20 18:02:02.57322645 +0000 UTC m=+850.511504740" Jul 20 18:02:03.001299 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.001264 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:03.002259 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.002236 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:02:03.148216 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.148187 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:03.151477 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.151459 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.155493 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.155469 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-559lz\"" Jul 20 18:02:03.155595 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.155469 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 18:02:03.162590 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.162567 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:03.243689 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.243653 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:03.327800 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.327730 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.327931 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.327804 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4z2\" (UniqueName: \"kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.428551 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.428513 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4z2\" (UniqueName: \"kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.428753 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.428566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.429179 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.429162 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.438232 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.438209 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4z2\" (UniqueName: \"kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2\") pod \"limitador-limitador-64c94fd64f-wltp2\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.462110 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.462077 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:03.603658 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:03.603617 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:03.605280 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:03.605242 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351964f8_0e9f_42af_80f4_fd7ef4844d1c.slice/crio-39a1acde04dac275b0d061670cfa3370b3e261e3043285319fd4cbcc6c796d23 WatchSource:0}: Error finding container 39a1acde04dac275b0d061670cfa3370b3e261e3043285319fd4cbcc6c796d23: Status 404 returned error can't find the container with id 39a1acde04dac275b0d061670cfa3370b3e261e3043285319fd4cbcc6c796d23 Jul 20 18:02:04.006545 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.006512 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" event={"ID":"351964f8-0e9f-42af-80f4-fd7ef4844d1c","Type":"ContainerStarted","Data":"39a1acde04dac275b0d061670cfa3370b3e261e3043285319fd4cbcc6c796d23"} Jul 20 18:02:04.084646 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.084600 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:04.089036 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.089019 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:04.093513 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.093490 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8dzr9\"" Jul 20 18:02:04.104041 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.104017 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:04.234089 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.234060 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7dc\" (UniqueName: \"kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc\") pod \"authorino-58cd955675-vblpz\" (UID: \"9bf2b932-4903-44c8-9c29-4191bdfd6d12\") " pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:04.334791 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.334724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7dc\" (UniqueName: \"kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc\") pod \"authorino-58cd955675-vblpz\" (UID: \"9bf2b932-4903-44c8-9c29-4191bdfd6d12\") " pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:04.346880 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.346855 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7dc\" (UniqueName: \"kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc\") pod \"authorino-58cd955675-vblpz\" (UID: \"9bf2b932-4903-44c8-9c29-4191bdfd6d12\") " pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:04.363012 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.362984 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:04.365943 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.365924 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:04.375539 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.375515 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:04.398721 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.398697 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:04.536569 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.536294 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5pf\" (UniqueName: \"kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf\") pod \"authorino-75d556bd77-qbzdr\" (UID: \"24a485e8-95d7-4c6c-b7fe-3a59bed4776f\") " pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:04.558121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.557174 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:04.637884 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.637805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5pf\" (UniqueName: \"kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf\") pod \"authorino-75d556bd77-qbzdr\" (UID: \"24a485e8-95d7-4c6c-b7fe-3a59bed4776f\") " pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:04.653296 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.653241 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5pf\" (UniqueName: \"kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf\") pod \"authorino-75d556bd77-qbzdr\" (UID: \"24a485e8-95d7-4c6c-b7fe-3a59bed4776f\") " pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:04.675873 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.675701 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:04.852942 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:04.852818 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:04.855919 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:04.855887 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24a485e8_95d7_4c6c_b7fe_3a59bed4776f.slice/crio-091b8189365a94a5a27500636f09b7c1c9b1733a5c5b755bf98960a3f8fbf5d3 WatchSource:0}: Error finding container 091b8189365a94a5a27500636f09b7c1c9b1733a5c5b755bf98960a3f8fbf5d3: Status 404 returned error can't find the container with id 091b8189365a94a5a27500636f09b7c1c9b1733a5c5b755bf98960a3f8fbf5d3 Jul 20 18:02:05.013949 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:05.013898 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qbzdr" event={"ID":"24a485e8-95d7-4c6c-b7fe-3a59bed4776f","Type":"ContainerStarted","Data":"091b8189365a94a5a27500636f09b7c1c9b1733a5c5b755bf98960a3f8fbf5d3"} Jul 20 18:02:05.015523 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:05.015488 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vblpz" event={"ID":"9bf2b932-4903-44c8-9c29-4191bdfd6d12","Type":"ContainerStarted","Data":"70050d2ec17fa98ec490c50304693ed8ce689988ce97b7c23b9923894e701ff7"} Jul 20 18:02:07.033466 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:07.033326 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" event={"ID":"351964f8-0e9f-42af-80f4-fd7ef4844d1c","Type":"ContainerStarted","Data":"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb"} Jul 20 18:02:07.033930 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:07.033706 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:10.048472 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.048435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qbzdr" event={"ID":"24a485e8-95d7-4c6c-b7fe-3a59bed4776f","Type":"ContainerStarted","Data":"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696"} Jul 20 18:02:10.049993 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.049967 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vblpz" event={"ID":"9bf2b932-4903-44c8-9c29-4191bdfd6d12","Type":"ContainerStarted","Data":"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8"} Jul 20 18:02:10.066103 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.066055 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-qbzdr" podStartSLOduration=1.839465637 podStartE2EDuration="6.066040676s" totalImagesPullingTime="4.226575039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:10.06597637 +0000 UTC m=+858.004254661" watchObservedRunningTime="2026-07-20 18:02:10.066040676 +0000 UTC m=+858.004318965" Jul 20 18:02:10.066951 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.066916 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" podStartSLOduration=3.94084192 podStartE2EDuration="7.066904248s" totalImagesPullingTime="3.126062328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:07.070525173 +0000 UTC m=+855.008803466" watchObservedRunningTime="2026-07-20 18:02:10.066904248 +0000 UTC m=+858.005182546" Jul 20 18:02:10.086274 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.086229 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-vblpz" podStartSLOduration=0.788949595 podStartE2EDuration="6.086202577s" totalImagesPullingTime="5.297252982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:10.085652036 +0000 UTC m=+858.023930327" watchObservedRunningTime="2026-07-20 18:02:10.086202577 +0000 UTC m=+858.024480868" Jul 20 18:02:10.117882 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:10.117813 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:12.058690 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:12.058648 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-vblpz" podUID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" containerName="authorino" containerID="cri-o://64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8" gracePeriod=30 Jul 20 18:02:12.313584 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:12.313525 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:12.402683 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:12.402651 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7dc\" (UniqueName: \"kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc\") pod \"9bf2b932-4903-44c8-9c29-4191bdfd6d12\" (UID: \"9bf2b932-4903-44c8-9c29-4191bdfd6d12\") " Jul 20 18:02:12.404636 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:12.404605 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc" (OuterVolumeSpecName: "kube-api-access-rb7dc") pod "9bf2b932-4903-44c8-9c29-4191bdfd6d12" (UID: "9bf2b932-4903-44c8-9c29-4191bdfd6d12"). InnerVolumeSpecName "kube-api-access-rb7dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:02:12.503165 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:12.503141 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7dc\" (UniqueName: \"kubernetes.io/projected/9bf2b932-4903-44c8-9c29-4191bdfd6d12-kube-api-access-rb7dc\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:13.063769 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.063731 2554 generic.go:356] "Generic (PLEG): container finished" podID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" containerID="64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8" exitCode=0 Jul 20 18:02:13.063769 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.063769 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vblpz" Jul 20 18:02:13.064263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.063791 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vblpz" event={"ID":"9bf2b932-4903-44c8-9c29-4191bdfd6d12","Type":"ContainerDied","Data":"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8"} Jul 20 18:02:13.064263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.063830 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vblpz" event={"ID":"9bf2b932-4903-44c8-9c29-4191bdfd6d12","Type":"ContainerDied","Data":"70050d2ec17fa98ec490c50304693ed8ce689988ce97b7c23b9923894e701ff7"} Jul 20 18:02:13.064263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.063853 2554 scope.go:117] "RemoveContainer" containerID="64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8" Jul 20 18:02:13.071925 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.071909 2554 scope.go:117] "RemoveContainer" containerID="64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8" Jul 20 18:02:13.072173 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:02:13.072151 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8\": container with ID starting with 64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8 not found: ID does not exist" containerID="64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8" Jul 20 18:02:13.072235 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.072185 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8"} err="failed to get container status \"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8\": rpc error: code = NotFound desc = could not find container \"64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8\": container with ID starting with 64615ac1791225a5e4910661571c159c52d255416bcded0bd41df6e04e3bbca8 not found: ID does not exist" Jul 20 18:02:13.081004 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.080983 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:13.084435 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:13.084417 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-vblpz"] Jul 20 18:02:14.549115 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:14.549085 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" path="/var/lib/kubelet/pods/9bf2b932-4903-44c8-9c29-4191bdfd6d12/volumes" Jul 20 18:02:18.039581 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:18.039544 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:18.530464 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:18.530430 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:18.530750 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:18.530702 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" podUID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" containerName="limitador" containerID="cri-o://6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb" gracePeriod=30 Jul 20 18:02:19.078277 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.078257 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:19.091416 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.091364 2554 generic.go:356] "Generic (PLEG): container finished" podID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" containerID="6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb" exitCode=0 Jul 20 18:02:19.091416 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.091402 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" Jul 20 18:02:19.091578 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.091445 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" event={"ID":"351964f8-0e9f-42af-80f4-fd7ef4844d1c","Type":"ContainerDied","Data":"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb"} Jul 20 18:02:19.091578 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.091506 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wltp2" event={"ID":"351964f8-0e9f-42af-80f4-fd7ef4844d1c","Type":"ContainerDied","Data":"39a1acde04dac275b0d061670cfa3370b3e261e3043285319fd4cbcc6c796d23"} Jul 20 18:02:19.091578 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.091536 2554 scope.go:117] "RemoveContainer" containerID="6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb" Jul 20 18:02:19.099971 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.099945 2554 scope.go:117] "RemoveContainer" containerID="6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb" Jul 20 18:02:19.100254 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:02:19.100223 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb\": container with ID starting with 6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb not found: ID does not exist" containerID="6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb" Jul 20 18:02:19.100340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.100257 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb"} err="failed to get container status \"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb\": rpc error: code = NotFound desc = could not find container \"6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb\": container with ID starting with 6f437e7a9d3115432d845d2234725733c00030e580b0bdc8c83e4a29d99a28bb not found: ID does not exist" Jul 20 18:02:19.157278 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.157253 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np4z2\" (UniqueName: \"kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2\") pod \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " Jul 20 18:02:19.157374 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.157362 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file\") pod \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\" (UID: \"351964f8-0e9f-42af-80f4-fd7ef4844d1c\") " Jul 20 18:02:19.157702 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.157683 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file" (OuterVolumeSpecName: "config-file") pod "351964f8-0e9f-42af-80f4-fd7ef4844d1c" (UID: "351964f8-0e9f-42af-80f4-fd7ef4844d1c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:02:19.159148 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.159126 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2" (OuterVolumeSpecName: "kube-api-access-np4z2") pod "351964f8-0e9f-42af-80f4-fd7ef4844d1c" (UID: "351964f8-0e9f-42af-80f4-fd7ef4844d1c"). InnerVolumeSpecName "kube-api-access-np4z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:02:19.257973 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.257949 2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/351964f8-0e9f-42af-80f4-fd7ef4844d1c-config-file\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:19.257973 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.257971 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4z2\" (UniqueName: \"kubernetes.io/projected/351964f8-0e9f-42af-80f4-fd7ef4844d1c-kube-api-access-np4z2\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:19.414234 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.414208 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:19.420149 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:19.420121 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wltp2"] Jul 20 18:02:20.553973 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:20.553941 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" path="/var/lib/kubelet/pods/351964f8-0e9f-42af-80f4-fd7ef4844d1c/volumes" Jul 20 18:02:24.810845 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.810784 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-48qpj"] Jul 20 18:02:24.811291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811193 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" containerName="limitador" Jul 20 18:02:24.811291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811207 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" containerName="limitador" Jul 20 18:02:24.811291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811222 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" containerName="authorino" Jul 20 18:02:24.811291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811228 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" containerName="authorino" Jul 20 18:02:24.811291 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811290 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="351964f8-0e9f-42af-80f4-fd7ef4844d1c" containerName="limitador" Jul 20 18:02:24.811477 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.811299 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf2b932-4903-44c8-9c29-4191bdfd6d12" containerName="authorino" Jul 20 18:02:24.815496 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.815474 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:24.818915 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.818893 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nvv8g\"" Jul 20 18:02:24.819026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.818962 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 18:02:24.820092 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.820071 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 18:02:24.820194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.820123 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 20 18:02:24.829642 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.829612 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-48qpj"] Jul 20 18:02:24.902287 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.902260 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/623215b7-9eae-4038-8a3c-7f6473161759-data\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:24.902384 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:24.902329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9pk\" (UniqueName: \"kubernetes.io/projected/623215b7-9eae-4038-8a3c-7f6473161759-kube-api-access-kt9pk\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.003601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.003573 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9pk\" (UniqueName: \"kubernetes.io/projected/623215b7-9eae-4038-8a3c-7f6473161759-kube-api-access-kt9pk\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.003722 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.003654 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/623215b7-9eae-4038-8a3c-7f6473161759-data\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.004044 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.004027 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/623215b7-9eae-4038-8a3c-7f6473161759-data\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.013519 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.013494 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9pk\" (UniqueName: \"kubernetes.io/projected/623215b7-9eae-4038-8a3c-7f6473161759-kube-api-access-kt9pk\") pod \"postgres-69fb4594cf-48qpj\" (UID: \"623215b7-9eae-4038-8a3c-7f6473161759\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.125110 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.125038 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:25.253404 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:25.253376 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-48qpj"] Jul 20 18:02:25.255326 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:25.255298 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623215b7_9eae_4038_8a3c_7f6473161759.slice/crio-9defdaf43f4c5a6c93c5fab270545aacc973e70034dfccd52bef18fc408d12bd WatchSource:0}: Error finding container 9defdaf43f4c5a6c93c5fab270545aacc973e70034dfccd52bef18fc408d12bd: Status 404 returned error can't find the container with id 9defdaf43f4c5a6c93c5fab270545aacc973e70034dfccd52bef18fc408d12bd Jul 20 18:02:26.129232 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:26.129189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" event={"ID":"623215b7-9eae-4038-8a3c-7f6473161759","Type":"ContainerStarted","Data":"9defdaf43f4c5a6c93c5fab270545aacc973e70034dfccd52bef18fc408d12bd"} Jul 20 18:02:31.155902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:31.155866 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" event={"ID":"623215b7-9eae-4038-8a3c-7f6473161759","Type":"ContainerStarted","Data":"9ba3f2484e63a60f7c4d2284552c5026bbeeeca21253b5f93307795215ebb627"} Jul 20 18:02:31.156295 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:31.156013 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:31.173477 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:31.173433 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" podStartSLOduration=2.138930724 podStartE2EDuration="7.173418998s" totalImagesPullingTime="5.034488274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:31.173134508 +0000 UTC m=+879.111412809" watchObservedRunningTime="2026-07-20 18:02:31.173418998 +0000 UTC m=+879.111697287" Jul 20 18:02:37.188883 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:37.188852 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-48qpj" Jul 20 18:02:37.768059 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:37.768027 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:37.775325 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:37.775303 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:37.781642 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:37.781597 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:37.924392 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:37.924363 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzgm\" (UniqueName: \"kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm\") pod \"authorino-5d4b67c446-d4j6h\" (UID: \"2b8b243f-842f-4aa8-81a2-f875897c0379\") " pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:38.025695 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.025592 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzgm\" (UniqueName: \"kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm\") pod \"authorino-5d4b67c446-d4j6h\" (UID: \"2b8b243f-842f-4aa8-81a2-f875897c0379\") " pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:38.049982 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.049955 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzgm\" (UniqueName: \"kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm\") pod \"authorino-5d4b67c446-d4j6h\" (UID: \"2b8b243f-842f-4aa8-81a2-f875897c0379\") " pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:38.063840 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.063815 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:38.064044 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.064032 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:38.087161 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.087133 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:38.091994 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.091976 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:38.102719 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.102692 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:38.202465 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.202443 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:38.204550 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:38.204527 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b8b243f_842f_4aa8_81a2_f875897c0379.slice/crio-ddc7c42942de2a9b6b0195190689c6c39eb83c297d926ebf1f3e530b634adac2 WatchSource:0}: Error finding container ddc7c42942de2a9b6b0195190689c6c39eb83c297d926ebf1f3e530b634adac2: Status 404 returned error can't find the container with id ddc7c42942de2a9b6b0195190689c6c39eb83c297d926ebf1f3e530b634adac2 Jul 20 18:02:38.228036 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.228009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zzt\" (UniqueName: \"kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt\") pod \"authorino-8d6988fc5-6lhnr\" (UID: \"d68ca14a-883c-498d-bc8f-7805ec7cbf33\") " pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:38.329348 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.329283 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zzt\" (UniqueName: \"kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt\") pod \"authorino-8d6988fc5-6lhnr\" (UID: \"d68ca14a-883c-498d-bc8f-7805ec7cbf33\") " pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:38.339282 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.339255 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zzt\" (UniqueName: \"kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt\") pod \"authorino-8d6988fc5-6lhnr\" (UID: \"d68ca14a-883c-498d-bc8f-7805ec7cbf33\") " pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:38.371238 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.371211 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:38.371448 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.371436 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:38.406984 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.406949 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:02:38.411517 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.411497 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.414850 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.414833 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 18:02:38.420617 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.420586 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:02:38.516615 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.516588 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:38.518657 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:38.518610 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd68ca14a_883c_498d_bc8f_7805ec7cbf33.slice/crio-d9e0008ea040e98114b4deef71524864692c7a75071660d70ba5d42a0bf96895 WatchSource:0}: Error finding container d9e0008ea040e98114b4deef71524864692c7a75071660d70ba5d42a0bf96895: Status 404 returned error can't find the container with id d9e0008ea040e98114b4deef71524864692c7a75071660d70ba5d42a0bf96895 Jul 20 18:02:38.530617 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.530596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlsn\" (UniqueName: \"kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.530741 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.530658 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.631954 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.631930 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlsn\" (UniqueName: \"kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.632052 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.631986 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.634207 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.634183 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.642072 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.642045 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlsn\" (UniqueName: \"kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn\") pod \"authorino-57f76c679-8f6rk\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.723402 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.723368 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:02:38.878019 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:38.877995 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:02:38.879917 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:38.879892 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd6bdd4_e89e_4249_a047_9fd171492a61.slice/crio-5d5deae2678dc5fcad8e6536f66a73545645cab8000841e251d6f0a553e7a749 WatchSource:0}: Error finding container 5d5deae2678dc5fcad8e6536f66a73545645cab8000841e251d6f0a553e7a749: Status 404 returned error can't find the container with id 5d5deae2678dc5fcad8e6536f66a73545645cab8000841e251d6f0a553e7a749 Jul 20 18:02:39.192143 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.192060 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" event={"ID":"d68ca14a-883c-498d-bc8f-7805ec7cbf33","Type":"ContainerStarted","Data":"d9e0008ea040e98114b4deef71524864692c7a75071660d70ba5d42a0bf96895"} Jul 20 18:02:39.193298 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.193271 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f76c679-8f6rk" event={"ID":"bdd6bdd4-e89e-4249-a047-9fd171492a61","Type":"ContainerStarted","Data":"5d5deae2678dc5fcad8e6536f66a73545645cab8000841e251d6f0a553e7a749"} Jul 20 18:02:39.194597 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.194579 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" event={"ID":"2b8b243f-842f-4aa8-81a2-f875897c0379","Type":"ContainerStarted","Data":"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251"} Jul 20 18:02:39.194710 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.194605 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" event={"ID":"2b8b243f-842f-4aa8-81a2-f875897c0379","Type":"ContainerStarted","Data":"ddc7c42942de2a9b6b0195190689c6c39eb83c297d926ebf1f3e530b634adac2"} Jul 20 18:02:39.194710 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.194661 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" podUID="2b8b243f-842f-4aa8-81a2-f875897c0379" containerName="authorino" containerID="cri-o://3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251" gracePeriod=30 Jul 20 18:02:39.211859 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.211815 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" podStartSLOduration=1.81144094 podStartE2EDuration="2.21180097s" totalImagesPullingTime="400.36003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:39.210826096 +0000 UTC m=+887.149104386" watchObservedRunningTime="2026-07-20 18:02:39.21180097 +0000 UTC m=+887.150079259" Jul 20 18:02:39.442837 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.442781 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:39.540271 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.540242 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spzgm\" (UniqueName: \"kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm\") pod \"2b8b243f-842f-4aa8-81a2-f875897c0379\" (UID: \"2b8b243f-842f-4aa8-81a2-f875897c0379\") " Jul 20 18:02:39.542303 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.542276 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm" (OuterVolumeSpecName: "kube-api-access-spzgm") pod "2b8b243f-842f-4aa8-81a2-f875897c0379" (UID: "2b8b243f-842f-4aa8-81a2-f875897c0379"). InnerVolumeSpecName "kube-api-access-spzgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:02:39.641819 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:39.641788 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spzgm\" (UniqueName: \"kubernetes.io/projected/2b8b243f-842f-4aa8-81a2-f875897c0379-kube-api-access-spzgm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:40.199689 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.199586 2554 generic.go:356] "Generic (PLEG): container finished" podID="2b8b243f-842f-4aa8-81a2-f875897c0379" containerID="3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251" exitCode=0 Jul 20 18:02:40.199689 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.199648 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" Jul 20 18:02:40.199689 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.199676 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" event={"ID":"2b8b243f-842f-4aa8-81a2-f875897c0379","Type":"ContainerDied","Data":"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251"} Jul 20 18:02:40.199940 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.199718 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-d4j6h" event={"ID":"2b8b243f-842f-4aa8-81a2-f875897c0379","Type":"ContainerDied","Data":"ddc7c42942de2a9b6b0195190689c6c39eb83c297d926ebf1f3e530b634adac2"} Jul 20 18:02:40.199940 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.199735 2554 scope.go:117] "RemoveContainer" containerID="3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251" Jul 20 18:02:40.207874 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.207855 2554 scope.go:117] "RemoveContainer" containerID="3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251" Jul 20 18:02:40.208129 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:02:40.208110 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251\": container with ID starting with 3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251 not found: ID does not exist" containerID="3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251" Jul 20 18:02:40.208184 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.208138 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251"} err="failed to get container status \"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251\": rpc error: code = NotFound desc = could not find container \"3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251\": container with ID starting with 3f3e730607df94bae437973809ea2c3059fa5f10a84769c19f10ab0a986f9251 not found: ID does not exist" Jul 20 18:02:40.220668 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.220642 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:40.229620 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.229598 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d4j6h"] Jul 20 18:02:40.555016 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:40.551274 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b8b243f-842f-4aa8-81a2-f875897c0379" path="/var/lib/kubelet/pods/2b8b243f-842f-4aa8-81a2-f875897c0379/volumes" Jul 20 18:02:42.210943 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.210849 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f76c679-8f6rk" event={"ID":"bdd6bdd4-e89e-4249-a047-9fd171492a61","Type":"ContainerStarted","Data":"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537"} Jul 20 18:02:42.212243 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.212218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" event={"ID":"d68ca14a-883c-498d-bc8f-7805ec7cbf33","Type":"ContainerStarted","Data":"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a"} Jul 20 18:02:42.212335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.212271 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" podUID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" containerName="authorino" containerID="cri-o://67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a" gracePeriod=30 Jul 20 18:02:42.237759 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.237711 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57f76c679-8f6rk" podStartSLOduration=1.200565879 podStartE2EDuration="4.237696327s" totalImagesPullingTime="3.037130448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:42.236153812 +0000 UTC m=+890.174432120" watchObservedRunningTime="2026-07-20 18:02:42.237696327 +0000 UTC m=+890.175974617" Jul 20 18:02:42.256366 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.256321 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" podStartSLOduration=1.314290039 podStartE2EDuration="4.256304401s" totalImagesPullingTime="2.942014362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:42.255831252 +0000 UTC m=+890.194109545" watchObservedRunningTime="2026-07-20 18:02:42.256304401 +0000 UTC m=+890.194582691" Jul 20 18:02:42.275386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.275356 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:42.275582 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.275559 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-qbzdr" podUID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" containerName="authorino" containerID="cri-o://fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696" gracePeriod=30 Jul 20 18:02:42.563155 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.563130 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:42.573183 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.573163 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:42.665613 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.665587 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p5pf\" (UniqueName: \"kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf\") pod \"24a485e8-95d7-4c6c-b7fe-3a59bed4776f\" (UID: \"24a485e8-95d7-4c6c-b7fe-3a59bed4776f\") " Jul 20 18:02:42.665814 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.665672 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8zzt\" (UniqueName: \"kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt\") pod \"d68ca14a-883c-498d-bc8f-7805ec7cbf33\" (UID: \"d68ca14a-883c-498d-bc8f-7805ec7cbf33\") " Jul 20 18:02:42.667578 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.667553 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf" (OuterVolumeSpecName: "kube-api-access-8p5pf") pod "24a485e8-95d7-4c6c-b7fe-3a59bed4776f" (UID: "24a485e8-95d7-4c6c-b7fe-3a59bed4776f"). InnerVolumeSpecName "kube-api-access-8p5pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:02:42.667716 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.667699 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt" (OuterVolumeSpecName: "kube-api-access-b8zzt") pod "d68ca14a-883c-498d-bc8f-7805ec7cbf33" (UID: "d68ca14a-883c-498d-bc8f-7805ec7cbf33"). InnerVolumeSpecName "kube-api-access-b8zzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:02:42.766442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.766415 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zzt\" (UniqueName: \"kubernetes.io/projected/d68ca14a-883c-498d-bc8f-7805ec7cbf33-kube-api-access-b8zzt\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:42.766442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:42.766438 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5pf\" (UniqueName: \"kubernetes.io/projected/24a485e8-95d7-4c6c-b7fe-3a59bed4776f-kube-api-access-8p5pf\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:02:43.217920 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.217833 2554 generic.go:356] "Generic (PLEG): container finished" podID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" containerID="67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a" exitCode=0 Jul 20 18:02:43.217920 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.217874 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" Jul 20 18:02:43.218415 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.217929 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" event={"ID":"d68ca14a-883c-498d-bc8f-7805ec7cbf33","Type":"ContainerDied","Data":"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a"} Jul 20 18:02:43.218415 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.217973 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d6988fc5-6lhnr" event={"ID":"d68ca14a-883c-498d-bc8f-7805ec7cbf33","Type":"ContainerDied","Data":"d9e0008ea040e98114b4deef71524864692c7a75071660d70ba5d42a0bf96895"} Jul 20 18:02:43.218415 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.217995 2554 scope.go:117] "RemoveContainer" containerID="67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a" Jul 20 18:02:43.219114 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.219096 2554 generic.go:356] "Generic (PLEG): container finished" podID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" containerID="fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696" exitCode=0 Jul 20 18:02:43.219222 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.219140 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qbzdr" Jul 20 18:02:43.219222 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.219188 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qbzdr" event={"ID":"24a485e8-95d7-4c6c-b7fe-3a59bed4776f","Type":"ContainerDied","Data":"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696"} Jul 20 18:02:43.219329 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.219227 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qbzdr" event={"ID":"24a485e8-95d7-4c6c-b7fe-3a59bed4776f","Type":"ContainerDied","Data":"091b8189365a94a5a27500636f09b7c1c9b1733a5c5b755bf98960a3f8fbf5d3"} Jul 20 18:02:43.226410 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.226393 2554 scope.go:117] "RemoveContainer" containerID="67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a" Jul 20 18:02:43.226688 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:02:43.226664 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a\": container with ID starting with 67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a not found: ID does not exist" containerID="67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a" Jul 20 18:02:43.226767 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.226699 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a"} err="failed to get container status \"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a\": rpc error: code = NotFound desc = could not find container \"67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a\": container with ID starting with 67a591c1f746eab18f8fc6eab1afceb076ebd432b1ae142b5b819ed8916e383a not found: ID does not exist" Jul 20 18:02:43.226767 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.226722 2554 scope.go:117] "RemoveContainer" containerID="fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696" Jul 20 18:02:43.234247 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.234232 2554 scope.go:117] "RemoveContainer" containerID="fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696" Jul 20 18:02:43.234483 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:02:43.234467 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696\": container with ID starting with fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696 not found: ID does not exist" containerID="fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696" Jul 20 18:02:43.234538 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.234488 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696"} err="failed to get container status \"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696\": rpc error: code = NotFound desc = could not find container \"fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696\": container with ID starting with fdf9f33e73bb40829feb1d500d525f3c6c9bd3fc7c6781328604db94e9368696 not found: ID does not exist" Jul 20 18:02:43.243990 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.243968 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:43.248376 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.248356 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qbzdr"] Jul 20 18:02:43.260205 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.260187 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:43.262474 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:43.262454 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8d6988fc5-6lhnr"] Jul 20 18:02:44.549187 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:44.549153 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" path="/var/lib/kubelet/pods/24a485e8-95d7-4c6c-b7fe-3a59bed4776f/volumes" Jul 20 18:02:44.549537 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:44.549458 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" path="/var/lib/kubelet/pods/d68ca14a-883c-498d-bc8f-7805ec7cbf33/volumes" Jul 20 18:02:47.346525 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346493 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346820 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" containerName="authorino" Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346832 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" containerName="authorino" Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346843 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" containerName="authorino" Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346848 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" containerName="authorino" Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346857 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8b243f-842f-4aa8-81a2-f875897c0379" containerName="authorino" Jul 20 18:02:47.346902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346862 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8b243f-842f-4aa8-81a2-f875897c0379" containerName="authorino" Jul 20 18:02:47.347084 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346912 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a485e8-95d7-4c6c-b7fe-3a59bed4776f" containerName="authorino" Jul 20 18:02:47.347084 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346922 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8b243f-842f-4aa8-81a2-f875897c0379" containerName="authorino" Jul 20 18:02:47.347084 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.346929 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68ca14a-883c-498d-bc8f-7805ec7cbf33" containerName="authorino" Jul 20 18:02:47.350716 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.350695 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.357800 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.357771 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mk6b9\"" Jul 20 18:02:47.358601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.358578 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 18:02:47.359009 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.358822 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 18:02:47.371618 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.371594 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:02:47.505651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.505601 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.505822 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.505666 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2c5\" (UniqueName: \"kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.540338 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.540299 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:02:47.543382 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.543361 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.554095 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.554062 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:02:47.606924 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.606848 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2c5\" (UniqueName: \"kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.607059 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.606949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.609305 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.609279 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.616137 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.616112 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2c5\" (UniqueName: \"kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5\") pod \"maas-controller-bd64cb5c-w4qrc\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.663213 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.663185 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:47.707961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.707929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.708098 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.708029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25j2p\" (UniqueName: \"kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.789198 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.789176 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:02:47.791450 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:47.791419 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32fe238_ad78_495f_96ab_5b8637737ec9.slice/crio-7caeec9b270c1aabdc75fedf92ebccdc336b42589fd549025af85e2afd1a9e26 WatchSource:0}: Error finding container 7caeec9b270c1aabdc75fedf92ebccdc336b42589fd549025af85e2afd1a9e26: Status 404 returned error can't find the container with id 7caeec9b270c1aabdc75fedf92ebccdc336b42589fd549025af85e2afd1a9e26 Jul 20 18:02:47.809157 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.809136 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25j2p\" (UniqueName: \"kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.809252 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.809196 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.811429 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.811407 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.819967 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.819945 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25j2p\" (UniqueName: \"kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p\") pod \"maas-controller-584c885fff-tzr2l\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.853962 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.853937 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:47.987801 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:47.987618 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:02:47.989902 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:02:47.989871 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc378e8_c76a_42c2_b879_cd3b816f8268.slice/crio-bc550a6918e7446a2acd7d6bd0626ea4fbf307246f8512a5648e1a7ed2062c4c WatchSource:0}: Error finding container bc550a6918e7446a2acd7d6bd0626ea4fbf307246f8512a5648e1a7ed2062c4c: Status 404 returned error can't find the container with id bc550a6918e7446a2acd7d6bd0626ea4fbf307246f8512a5648e1a7ed2062c4c Jul 20 18:02:48.244604 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:48.244504 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" event={"ID":"b32fe238-ad78-495f-96ab-5b8637737ec9","Type":"ContainerStarted","Data":"7caeec9b270c1aabdc75fedf92ebccdc336b42589fd549025af85e2afd1a9e26"} Jul 20 18:02:48.245675 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:48.245643 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-tzr2l" event={"ID":"8dc378e8-c76a-42c2-b879-cd3b816f8268","Type":"ContainerStarted","Data":"bc550a6918e7446a2acd7d6bd0626ea4fbf307246f8512a5648e1a7ed2062c4c"} Jul 20 18:02:51.263103 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.263059 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" event={"ID":"b32fe238-ad78-495f-96ab-5b8637737ec9","Type":"ContainerStarted","Data":"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c"} Jul 20 18:02:51.263524 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.263242 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:02:51.264599 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.264572 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-tzr2l" event={"ID":"8dc378e8-c76a-42c2-b879-cd3b816f8268","Type":"ContainerStarted","Data":"fd192875e7a6acdc444c0c9d8f4e46d1cd3dd86a6a441fc2d786e7231b08f617"} Jul 20 18:02:51.264727 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.264664 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:02:51.283979 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.283938 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" podStartSLOduration=1.591807498 podStartE2EDuration="4.283926049s" totalImagesPullingTime="2.692118551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:51.281801323 +0000 UTC m=+899.220079624" watchObservedRunningTime="2026-07-20 18:02:51.283926049 +0000 UTC m=+899.222204340" Jul 20 18:02:51.301428 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:51.301382 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-584c885fff-tzr2l" podStartSLOduration=1.801086524 podStartE2EDuration="4.301368784s" totalImagesPullingTime="2.50028226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:51.299828005 +0000 UTC m=+899.238106319" watchObservedRunningTime="2026-07-20 18:02:51.301368784 +0000 UTC m=+899.239647073" Jul 20 18:02:52.467586 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:52.467556 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:02:52.468163 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:02:52.468141 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:03:02.272245 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.272212 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:03:02.272750 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.272300 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:03:02.344464 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.344425 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:03:02.344712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.344687 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" podUID="b32fe238-ad78-495f-96ab-5b8637737ec9" containerName="manager" containerID="cri-o://966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c" gracePeriod=30 Jul 20 18:03:02.592916 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.592891 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:03:02.634840 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.634810 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert\") pod \"b32fe238-ad78-495f-96ab-5b8637737ec9\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " Jul 20 18:03:02.635013 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.634870 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2c5\" (UniqueName: \"kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5\") pod \"b32fe238-ad78-495f-96ab-5b8637737ec9\" (UID: \"b32fe238-ad78-495f-96ab-5b8637737ec9\") " Jul 20 18:03:02.637081 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.637051 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5" (OuterVolumeSpecName: "kube-api-access-zn2c5") pod "b32fe238-ad78-495f-96ab-5b8637737ec9" (UID: "b32fe238-ad78-495f-96ab-5b8637737ec9"). InnerVolumeSpecName "kube-api-access-zn2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:03:02.637275 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.637251 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b32fe238-ad78-495f-96ab-5b8637737ec9" (UID: "b32fe238-ad78-495f-96ab-5b8637737ec9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:03:02.735702 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.735669 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b32fe238-ad78-495f-96ab-5b8637737ec9-webhook-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:03:02.735702 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:02.735696 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2c5\" (UniqueName: \"kubernetes.io/projected/b32fe238-ad78-495f-96ab-5b8637737ec9-kube-api-access-zn2c5\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:03:03.317210 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.317179 2554 generic.go:356] "Generic (PLEG): container finished" podID="b32fe238-ad78-495f-96ab-5b8637737ec9" containerID="966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c" exitCode=0 Jul 20 18:03:03.317740 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.317229 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" Jul 20 18:03:03.317740 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.317272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" event={"ID":"b32fe238-ad78-495f-96ab-5b8637737ec9","Type":"ContainerDied","Data":"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c"} Jul 20 18:03:03.317740 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.317325 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd64cb5c-w4qrc" event={"ID":"b32fe238-ad78-495f-96ab-5b8637737ec9","Type":"ContainerDied","Data":"7caeec9b270c1aabdc75fedf92ebccdc336b42589fd549025af85e2afd1a9e26"} Jul 20 18:03:03.317740 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.317347 2554 scope.go:117] "RemoveContainer" containerID="966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c" Jul 20 18:03:03.325942 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.325899 2554 scope.go:117] "RemoveContainer" containerID="966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c" Jul 20 18:03:03.326182 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:03:03.326163 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c\": container with ID starting with 966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c not found: ID does not exist" containerID="966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c" Jul 20 18:03:03.326261 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.326195 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c"} err="failed to get container status \"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c\": rpc error: code = NotFound desc = could not find container \"966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c\": container with ID starting with 966c708ce57af5b42b1bc06c6b1ed28c7e388f59d0b8571851a062af3edb923c not found: ID does not exist" Jul 20 18:03:03.340444 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.340418 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:03:03.346036 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:03.346013 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd64cb5c-w4qrc"] Jul 20 18:03:04.549481 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:04.549449 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32fe238-ad78-495f-96ab-5b8637737ec9" path="/var/lib/kubelet/pods/b32fe238-ad78-495f-96ab-5b8637737ec9/volumes" Jul 20 18:03:24.606679 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.606647 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-w9mr7"] Jul 20 18:03:24.607146 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.606955 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32fe238-ad78-495f-96ab-5b8637737ec9" containerName="manager" Jul 20 18:03:24.607146 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.606965 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32fe238-ad78-495f-96ab-5b8637737ec9" containerName="manager" Jul 20 18:03:24.607146 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.607036 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32fe238-ad78-495f-96ab-5b8637737ec9" containerName="manager" Jul 20 18:03:24.614593 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.614573 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.617873 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.617853 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xwr79\"" Jul 20 18:03:24.618911 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.618895 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 20 18:03:24.626727 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.626703 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-w9mr7"] Jul 20 18:03:24.639373 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.639349 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-nv7tm"] Jul 20 18:03:24.642915 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.642896 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.660688 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.660662 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-nv7tm"] Jul 20 18:03:24.706210 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.706176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.706349 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.706227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbf4\" (UniqueName: \"kubernetes.io/projected/3f02a0cf-de11-4252-bc3f-30720bb552aa-kube-api-access-szbf4\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.706349 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.706260 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllgj\" (UniqueName: \"kubernetes.io/projected/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-kube-api-access-bllgj\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.706349 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.706281 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f02a0cf-de11-4252-bc3f-30720bb552aa-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.807018 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.806988 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.807208 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.807043 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbf4\" (UniqueName: \"kubernetes.io/projected/3f02a0cf-de11-4252-bc3f-30720bb552aa-kube-api-access-szbf4\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.807208 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.807075 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllgj\" (UniqueName: \"kubernetes.io/projected/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-kube-api-access-bllgj\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.807208 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.807103 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f02a0cf-de11-4252-bc3f-30720bb552aa-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.807815 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.807793 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f02a0cf-de11-4252-bc3f-30720bb552aa-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.807918 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.807815 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-plugins-config-volume\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.815670 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.815615 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbf4\" (UniqueName: \"kubernetes.io/projected/3f02a0cf-de11-4252-bc3f-30720bb552aa-kube-api-access-szbf4\") pod \"payload-pre-processing-7677b774bb-w9mr7\" (UID: \"3f02a0cf-de11-4252-bc3f-30720bb552aa\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.815747 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.815702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllgj\" (UniqueName: \"kubernetes.io/projected/cb9af6ee-cf84-4ab1-91d8-89547006f1ba-kube-api-access-bllgj\") pod \"payload-processing-5bf47cd988-nv7tm\" (UID: \"cb9af6ee-cf84-4ab1-91d8-89547006f1ba\") " pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:24.925726 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.925615 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:24.952322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:24.952300 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:25.073162 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:25.073132 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-w9mr7"] Jul 20 18:03:25.076360 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:03:25.076307 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f02a0cf_de11_4252_bc3f_30720bb552aa.slice/crio-f91eea55ca1ce28c3bf080b225604aa9434f8ac2abbb36106953e0abe0e529bc WatchSource:0}: Error finding container f91eea55ca1ce28c3bf080b225604aa9434f8ac2abbb36106953e0abe0e529bc: Status 404 returned error can't find the container with id f91eea55ca1ce28c3bf080b225604aa9434f8ac2abbb36106953e0abe0e529bc Jul 20 18:03:25.101830 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:25.101806 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-nv7tm"] Jul 20 18:03:25.103461 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:03:25.103436 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9af6ee_cf84_4ab1_91d8_89547006f1ba.slice/crio-aae8e3e5980415538f8ae0012c8f085c478df6f6d7f62aa31a570b23962b4b2a WatchSource:0}: Error finding container aae8e3e5980415538f8ae0012c8f085c478df6f6d7f62aa31a570b23962b4b2a: Status 404 returned error can't find the container with id aae8e3e5980415538f8ae0012c8f085c478df6f6d7f62aa31a570b23962b4b2a Jul 20 18:03:25.412252 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:25.412218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" event={"ID":"3f02a0cf-de11-4252-bc3f-30720bb552aa","Type":"ContainerStarted","Data":"f91eea55ca1ce28c3bf080b225604aa9434f8ac2abbb36106953e0abe0e529bc"} Jul 20 18:03:25.413307 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:25.413284 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" event={"ID":"cb9af6ee-cf84-4ab1-91d8-89547006f1ba","Type":"ContainerStarted","Data":"aae8e3e5980415538f8ae0012c8f085c478df6f6d7f62aa31a570b23962b4b2a"} Jul 20 18:03:28.447723 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.447602 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" event={"ID":"cb9af6ee-cf84-4ab1-91d8-89547006f1ba","Type":"ContainerStarted","Data":"c1e1947bb4fef1a59b660231fabc6905d1d48de9c0199c16468794756b295b63"} Jul 20 18:03:28.448461 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.448432 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:28.450341 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.450313 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" event={"ID":"3f02a0cf-de11-4252-bc3f-30720bb552aa","Type":"ContainerStarted","Data":"d6f14f4aa68341ff40bdea1b41656596fe448bcd801042f7dbdd4d41ecb2d3b0"} Jul 20 18:03:28.450825 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.450802 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:28.467468 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.467426 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" podStartSLOduration=1.4189731540000001 podStartE2EDuration="4.467414336s" totalImagesPullingTime="3.048441182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:28.465787657 +0000 UTC m=+936.404065951" watchObservedRunningTime="2026-07-20 18:03:28.467414336 +0000 UTC m=+936.405692626" Jul 20 18:03:28.483676 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:28.483617 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" podStartSLOduration=1.408815857 podStartE2EDuration="4.48360202s" totalImagesPullingTime="3.074786163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:28.482324435 +0000 UTC m=+936.420602751" watchObservedRunningTime="2026-07-20 18:03:28.48360202 +0000 UTC m=+936.421880309" Jul 20 18:03:32.470667 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.470536 2554 generic.go:356] "Generic (PLEG): container finished" podID="ee786b93-3864-4931-8670-b7b1736db018" containerID="1442851e1463bbf5cb2aed5b90f57dffe46a1c4266e6a91f470259ffc235e754" exitCode=0 Jul 20 18:03:32.470667 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.470609 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" event={"ID":"ee786b93-3864-4931-8670-b7b1736db018","Type":"ContainerDied","Data":"1442851e1463bbf5cb2aed5b90f57dffe46a1c4266e6a91f470259ffc235e754"} Jul 20 18:03:32.471314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.471019 2554 scope.go:117] "RemoveContainer" containerID="1442851e1463bbf5cb2aed5b90f57dffe46a1c4266e6a91f470259ffc235e754" Jul 20 18:03:32.471997 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.471975 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:03:32.472047 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.472037 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:03:32.472081 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:32.472072 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:03:33.001487 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:33.001458 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:03:33.477553 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:33.477475 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" event={"ID":"ee786b93-3864-4931-8670-b7b1736db018","Type":"ContainerStarted","Data":"aab3760d98de7bdd9630ed8f76376dc7039b220babc3405c04e9a48fe0ef16a8"} Jul 20 18:03:33.540233 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:33.540189 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:03:33.540432 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:33.540365 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-tlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 20 18:03:33.540432 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:33.540414 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" podUID="ee786b93-3864-4931-8670-b7b1736db018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 20 18:03:34.539807 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:34.539772 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-tlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 20 18:03:34.540347 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:34.540317 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" podUID="ee786b93-3864-4931-8670-b7b1736db018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 20 18:03:35.539472 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:35.539437 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-tlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 20 18:03:35.539782 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:35.539753 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" podUID="ee786b93-3864-4931-8670-b7b1736db018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 20 18:03:36.539521 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:36.539481 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-tlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 20 18:03:36.540157 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:36.540120 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" podUID="ee786b93-3864-4931-8670-b7b1736db018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 20 18:03:37.542768 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:37.542738 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:03:37.543204 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:37.543030 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:03:37.543729 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:37.543707 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-tlcgp" Jul 20 18:03:40.461788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:40.461759 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-nv7tm" Jul 20 18:03:40.462195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:40.461906 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-w9mr7" Jul 20 18:03:43.990426 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.990378 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh"] Jul 20 18:03:43.995739 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.995715 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:43.998492 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.998464 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 18:03:43.998637 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.998492 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 18:03:43.999582 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.999546 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kpjkh\"" Jul 20 18:03:43.999706 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:43.999546 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 20 18:03:44.004974 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.004949 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh"] Jul 20 18:03:44.091664 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.091831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.091831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091702 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.091831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.091831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091760 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21518261-e875-4fb8-b46c-84da91443053-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.091831 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.091799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5nl\" (UniqueName: \"kubernetes.io/projected/21518261-e875-4fb8-b46c-84da91443053-kube-api-access-pq5nl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.192955 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.192922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5nl\" (UniqueName: \"kubernetes.io/projected/21518261-e875-4fb8-b46c-84da91443053-kube-api-access-pq5nl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.192984 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193047 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193071 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193091 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21518261-e875-4fb8-b46c-84da91443053-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193408 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193387 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193466 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193418 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.193516 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.193487 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.195365 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.195342 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21518261-e875-4fb8-b46c-84da91443053-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.195590 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.195573 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21518261-e875-4fb8-b46c-84da91443053-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.201564 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.201534 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5nl\" (UniqueName: \"kubernetes.io/projected/21518261-e875-4fb8-b46c-84da91443053-kube-api-access-pq5nl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh\" (UID: \"21518261-e875-4fb8-b46c-84da91443053\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.307494 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.307459 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:44.450249 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.450157 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh"] Jul 20 18:03:44.452882 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:03:44.452853 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21518261_e875_4fb8_b46c_84da91443053.slice/crio-38e0bec3703c7e605cca510f767c8ce51028d5bc2b48cca85e4a99a033444b62 WatchSource:0}: Error finding container 38e0bec3703c7e605cca510f767c8ce51028d5bc2b48cca85e4a99a033444b62: Status 404 returned error can't find the container with id 38e0bec3703c7e605cca510f767c8ce51028d5bc2b48cca85e4a99a033444b62 Jul 20 18:03:44.454427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.454405 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 18:03:44.531682 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:44.531653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" event={"ID":"21518261-e875-4fb8-b46c-84da91443053","Type":"ContainerStarted","Data":"38e0bec3703c7e605cca510f767c8ce51028d5bc2b48cca85e4a99a033444b62"} Jul 20 18:03:46.543160 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:46.543124 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" event={"ID":"21518261-e875-4fb8-b46c-84da91443053","Type":"ContainerStarted","Data":"25b41d065fe516c6a3816a71afcb09bf3853280605a3e1dcb78378b928e9fdfc"} Jul 20 18:03:46.596682 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:46.596605 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" podStartSLOduration=2.3050128 podStartE2EDuration="3.596591212s" totalImagesPullingTime="1.291578412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:46.596087805 +0000 UTC m=+954.534366107" watchObservedRunningTime="2026-07-20 18:03:46.596591212 +0000 UTC m=+954.534869501" Jul 20 18:03:54.307944 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:54.307915 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:54.308429 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:54.308013 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:54.320298 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:54.320275 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:03:54.587762 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:03:54.587692 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh" Jul 20 18:04:14.986329 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:14.986253 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw"] Jul 20 18:04:14.993435 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:14.993414 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:14.996095 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:14.996073 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 20 18:04:15.000933 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.000909 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw"] Jul 20 18:04:15.135492 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135458 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.135663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135495 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdndl\" (UniqueName: \"kubernetes.io/projected/e270b02f-1cb6-4f79-9035-8dccbe789740-kube-api-access-jdndl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.135663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.135663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135611 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.135781 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.135781 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.135775 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270b02f-1cb6-4f79-9035-8dccbe789740-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236447 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236377 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236447 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236412 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270b02f-1cb6-4f79-9035-8dccbe789740-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236698 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236447 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236698 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236472 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdndl\" (UniqueName: \"kubernetes.io/projected/e270b02f-1cb6-4f79-9035-8dccbe789740-kube-api-access-jdndl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236698 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236505 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236698 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236552 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.236983 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236962 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.237046 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.236966 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.237046 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.237002 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.238659 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.238619 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e270b02f-1cb6-4f79-9035-8dccbe789740-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.238914 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.238897 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e270b02f-1cb6-4f79-9035-8dccbe789740-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.246427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.246403 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdndl\" (UniqueName: \"kubernetes.io/projected/e270b02f-1cb6-4f79-9035-8dccbe789740-kube-api-access-jdndl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw\" (UID: \"e270b02f-1cb6-4f79-9035-8dccbe789740\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.304934 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.304900 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:15.433429 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.433403 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw"] Jul 20 18:04:15.436174 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:04:15.436149 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode270b02f_1cb6_4f79_9035_8dccbe789740.slice/crio-6564aa25b7ad585b715c6eb75f486f8187d3136ad3b27f7e86ad33abead555e4 WatchSource:0}: Error finding container 6564aa25b7ad585b715c6eb75f486f8187d3136ad3b27f7e86ad33abead555e4: Status 404 returned error can't find the container with id 6564aa25b7ad585b715c6eb75f486f8187d3136ad3b27f7e86ad33abead555e4 Jul 20 18:04:15.678057 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:15.678029 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" event={"ID":"e270b02f-1cb6-4f79-9035-8dccbe789740","Type":"ContainerStarted","Data":"6564aa25b7ad585b715c6eb75f486f8187d3136ad3b27f7e86ad33abead555e4"} Jul 20 18:04:16.684206 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:16.684171 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" event={"ID":"e270b02f-1cb6-4f79-9035-8dccbe789740","Type":"ContainerStarted","Data":"e8e5830a4d82b7fa35d4694365939b7dc07be5b1366903ba777e59f6261f198f"} Jul 20 18:04:16.718363 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:16.718298 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" podStartSLOduration=2.517950301 podStartE2EDuration="2.718276925s" totalImagesPullingTime="200.326624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:04:16.715580184 +0000 UTC m=+984.653858493" watchObservedRunningTime="2026-07-20 18:04:16.718276925 +0000 UTC m=+984.656555216" Jul 20 18:04:22.291370 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.291333 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr"] Jul 20 18:04:22.297311 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.297288 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.299931 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.299906 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 20 18:04:22.304259 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.304232 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr"] Jul 20 18:04:22.396397 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396363 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.396588 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396405 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4fh\" (UniqueName: \"kubernetes.io/projected/928cfd54-ba59-40be-8c1c-2a3c614dee81-kube-api-access-zz4fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.396588 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396433 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.396588 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396482 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.396790 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396600 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928cfd54-ba59-40be-8c1c-2a3c614dee81-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.396790 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.396665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.497864 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497825 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4fh\" (UniqueName: \"kubernetes.io/projected/928cfd54-ba59-40be-8c1c-2a3c614dee81-kube-api-access-zz4fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928cfd54-ba59-40be-8c1c-2a3c614dee81-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497948 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.497994 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498348 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.498324 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498414 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.498348 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.498473 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.498406 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.500248 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.500224 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928cfd54-ba59-40be-8c1c-2a3c614dee81-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.500537 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.500507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928cfd54-ba59-40be-8c1c-2a3c614dee81-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.508657 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.508612 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4fh\" (UniqueName: \"kubernetes.io/projected/928cfd54-ba59-40be-8c1c-2a3c614dee81-kube-api-access-zz4fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2qmzr\" (UID: \"928cfd54-ba59-40be-8c1c-2a3c614dee81\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.608527 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.608450 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:22.741096 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:22.741071 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr"] Jul 20 18:04:22.743002 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:04:22.742976 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928cfd54_ba59_40be_8c1c_2a3c614dee81.slice/crio-c50f0651cdf78ea50a9f1c18453fde92a49d00a754c997ddb2c96f2a71616ec2 WatchSource:0}: Error finding container c50f0651cdf78ea50a9f1c18453fde92a49d00a754c997ddb2c96f2a71616ec2: Status 404 returned error can't find the container with id c50f0651cdf78ea50a9f1c18453fde92a49d00a754c997ddb2c96f2a71616ec2 Jul 20 18:04:23.718099 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:23.718068 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" event={"ID":"928cfd54-ba59-40be-8c1c-2a3c614dee81","Type":"ContainerStarted","Data":"4a58b8d5b08fa62daba8a84bcf8b6ba2c2c794bbad57ac3a7e3496df8e3c77c8"} Jul 20 18:04:23.718508 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:23.718118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" event={"ID":"928cfd54-ba59-40be-8c1c-2a3c614dee81","Type":"ContainerStarted","Data":"c50f0651cdf78ea50a9f1c18453fde92a49d00a754c997ddb2c96f2a71616ec2"} Jul 20 18:04:23.736455 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:23.736395 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" podStartSLOduration=1.5139591430000001 podStartE2EDuration="1.736380328s" totalImagesPullingTime="222.421185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:04:23.735664374 +0000 UTC m=+991.673942666" watchObservedRunningTime="2026-07-20 18:04:23.736380328 +0000 UTC m=+991.674658618" Jul 20 18:04:25.305693 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:25.305654 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:25.306056 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:25.305724 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:25.318365 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:25.318340 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:25.741651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:25.741533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw" Jul 20 18:04:32.608889 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:32.608859 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:32.609318 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:32.608912 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:32.621554 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:32.621533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:04:32.772125 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:04:32.772099 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2qmzr" Jul 20 18:05:27.789601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:27.789547 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b67867474-g8b4g"] Jul 20 18:05:27.793034 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:27.793016 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:27.801304 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:27.801284 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b67867474-g8b4g"] Jul 20 18:05:27.925755 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:27.925718 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2hq\" (UniqueName: \"kubernetes.io/projected/564a169a-4c89-4367-82f1-788b5cd2fa75-kube-api-access-dj2hq\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:27.925927 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:27.925801 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/564a169a-4c89-4367-82f1-788b5cd2fa75-tls-cert\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.026220 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.026189 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2hq\" (UniqueName: \"kubernetes.io/projected/564a169a-4c89-4367-82f1-788b5cd2fa75-kube-api-access-dj2hq\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.026378 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.026250 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/564a169a-4c89-4367-82f1-788b5cd2fa75-tls-cert\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.028768 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.028739 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/564a169a-4c89-4367-82f1-788b5cd2fa75-tls-cert\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.034285 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.034260 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2hq\" (UniqueName: \"kubernetes.io/projected/564a169a-4c89-4367-82f1-788b5cd2fa75-kube-api-access-dj2hq\") pod \"authorino-5b67867474-g8b4g\" (UID: \"564a169a-4c89-4367-82f1-788b5cd2fa75\") " pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.102373 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.102303 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b67867474-g8b4g" Jul 20 18:05:28.229519 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:28.229495 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b67867474-g8b4g"] Jul 20 18:05:28.231192 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:05:28.231162 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod564a169a_4c89_4367_82f1_788b5cd2fa75.slice/crio-9dafd5a2802dd949f795c5e8c7487ff7a8ab0abc2660f912e28b1b97827b7659 WatchSource:0}: Error finding container 9dafd5a2802dd949f795c5e8c7487ff7a8ab0abc2660f912e28b1b97827b7659: Status 404 returned error can't find the container with id 9dafd5a2802dd949f795c5e8c7487ff7a8ab0abc2660f912e28b1b97827b7659 Jul 20 18:05:29.022874 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.022834 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b67867474-g8b4g" event={"ID":"564a169a-4c89-4367-82f1-788b5cd2fa75","Type":"ContainerStarted","Data":"f2837cfcd450b7f246dd8c15967631f4a0c7376082ce13fd352be52ac674b089"} Jul 20 18:05:29.022874 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.022879 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b67867474-g8b4g" event={"ID":"564a169a-4c89-4367-82f1-788b5cd2fa75","Type":"ContainerStarted","Data":"9dafd5a2802dd949f795c5e8c7487ff7a8ab0abc2660f912e28b1b97827b7659"} Jul 20 18:05:29.040614 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.040550 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b67867474-g8b4g" podStartSLOduration=1.565232592 podStartE2EDuration="2.040528529s" totalImagesPullingTime="475.295937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:05:29.039354471 +0000 UTC m=+1056.977632763" watchObservedRunningTime="2026-07-20 18:05:29.040528529 +0000 UTC m=+1056.978806820" Jul 20 18:05:29.075201 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.075162 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:05:29.075407 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.075380 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-57f76c679-8f6rk" podUID="bdd6bdd4-e89e-4249-a047-9fd171492a61" containerName="authorino" containerID="cri-o://fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537" gracePeriod=30 Jul 20 18:05:29.340698 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.340671 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:05:29.438177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.438138 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert\") pod \"bdd6bdd4-e89e-4249-a047-9fd171492a61\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " Jul 20 18:05:29.438357 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.438241 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzlsn\" (UniqueName: \"kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn\") pod \"bdd6bdd4-e89e-4249-a047-9fd171492a61\" (UID: \"bdd6bdd4-e89e-4249-a047-9fd171492a61\") " Jul 20 18:05:29.440227 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.440195 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn" (OuterVolumeSpecName: "kube-api-access-tzlsn") pod "bdd6bdd4-e89e-4249-a047-9fd171492a61" (UID: "bdd6bdd4-e89e-4249-a047-9fd171492a61"). InnerVolumeSpecName "kube-api-access-tzlsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:05:29.449539 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.449509 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bdd6bdd4-e89e-4249-a047-9fd171492a61" (UID: "bdd6bdd4-e89e-4249-a047-9fd171492a61"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:05:29.539185 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.539139 2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdd6bdd4-e89e-4249-a047-9fd171492a61-tls-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:05:29.539185 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:29.539169 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzlsn\" (UniqueName: \"kubernetes.io/projected/bdd6bdd4-e89e-4249-a047-9fd171492a61-kube-api-access-tzlsn\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:05:30.029777 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.029737 2554 generic.go:356] "Generic (PLEG): container finished" podID="bdd6bdd4-e89e-4249-a047-9fd171492a61" containerID="fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537" exitCode=0 Jul 20 18:05:30.030212 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.029781 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f76c679-8f6rk" Jul 20 18:05:30.030212 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.029823 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f76c679-8f6rk" event={"ID":"bdd6bdd4-e89e-4249-a047-9fd171492a61","Type":"ContainerDied","Data":"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537"} Jul 20 18:05:30.030212 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.029868 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f76c679-8f6rk" event={"ID":"bdd6bdd4-e89e-4249-a047-9fd171492a61","Type":"ContainerDied","Data":"5d5deae2678dc5fcad8e6536f66a73545645cab8000841e251d6f0a553e7a749"} Jul 20 18:05:30.030212 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.029892 2554 scope.go:117] "RemoveContainer" containerID="fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537" Jul 20 18:05:30.044746 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.044726 2554 scope.go:117] "RemoveContainer" containerID="fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537" Jul 20 18:05:30.045025 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:05:30.045008 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537\": container with ID starting with fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537 not found: ID does not exist" containerID="fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537" Jul 20 18:05:30.045077 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.045034 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537"} err="failed to get container status \"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537\": rpc error: code = NotFound desc = could not find container \"fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537\": container with ID starting with fdb97028348052df0a8653eb9e2477fb0c63d96f9185ace0fca17ca9d029a537 not found: ID does not exist" Jul 20 18:05:30.063058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.063027 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:05:30.065613 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.065591 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f76c679-8f6rk"] Jul 20 18:05:30.550486 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:05:30.550443 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd6bdd4-e89e-4249-a047-9fd171492a61" path="/var/lib/kubelet/pods/bdd6bdd4-e89e-4249-a047-9fd171492a61/volumes" Jul 20 18:06:19.131897 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.131863 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:06:19.132459 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.132060 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-584c885fff-tzr2l" podUID="8dc378e8-c76a-42c2-b879-cd3b816f8268" containerName="manager" containerID="cri-o://fd192875e7a6acdc444c0c9d8f4e46d1cd3dd86a6a441fc2d786e7231b08f617" gracePeriod=30 Jul 20 18:06:19.269079 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.269040 2554 generic.go:356] "Generic (PLEG): container finished" podID="8dc378e8-c76a-42c2-b879-cd3b816f8268" containerID="fd192875e7a6acdc444c0c9d8f4e46d1cd3dd86a6a441fc2d786e7231b08f617" exitCode=0 Jul 20 18:06:19.269207 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.269110 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-tzr2l" event={"ID":"8dc378e8-c76a-42c2-b879-cd3b816f8268","Type":"ContainerDied","Data":"fd192875e7a6acdc444c0c9d8f4e46d1cd3dd86a6a441fc2d786e7231b08f617"} Jul 20 18:06:19.383844 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.383786 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:06:19.461684 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.461655 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert\") pod \"8dc378e8-c76a-42c2-b879-cd3b816f8268\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " Jul 20 18:06:19.461820 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.461715 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25j2p\" (UniqueName: \"kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p\") pod \"8dc378e8-c76a-42c2-b879-cd3b816f8268\" (UID: \"8dc378e8-c76a-42c2-b879-cd3b816f8268\") " Jul 20 18:06:19.463647 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.463588 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p" (OuterVolumeSpecName: "kube-api-access-25j2p") pod "8dc378e8-c76a-42c2-b879-cd3b816f8268" (UID: "8dc378e8-c76a-42c2-b879-cd3b816f8268"). InnerVolumeSpecName "kube-api-access-25j2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:06:19.463760 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.463706 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8dc378e8-c76a-42c2-b879-cd3b816f8268" (UID: "8dc378e8-c76a-42c2-b879-cd3b816f8268"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:06:19.562863 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.562836 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25j2p\" (UniqueName: \"kubernetes.io/projected/8dc378e8-c76a-42c2-b879-cd3b816f8268-kube-api-access-25j2p\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:06:19.562863 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:19.562860 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8dc378e8-c76a-42c2-b879-cd3b816f8268-webhook-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:06:20.275484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.275459 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-tzr2l" Jul 20 18:06:20.275484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.275478 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-tzr2l" event={"ID":"8dc378e8-c76a-42c2-b879-cd3b816f8268","Type":"ContainerDied","Data":"bc550a6918e7446a2acd7d6bd0626ea4fbf307246f8512a5648e1a7ed2062c4c"} Jul 20 18:06:20.275977 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.275529 2554 scope.go:117] "RemoveContainer" containerID="fd192875e7a6acdc444c0c9d8f4e46d1cd3dd86a6a441fc2d786e7231b08f617" Jul 20 18:06:20.296991 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.296968 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:06:20.300717 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.300697 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-584c885fff-tzr2l"] Jul 20 18:06:20.549992 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:06:20.549918 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc378e8-c76a-42c2-b879-cd3b816f8268" path="/var/lib/kubelet/pods/8dc378e8-c76a-42c2-b879-cd3b816f8268/volumes" Jul 20 18:07:21.166339 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166301 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:21.166798 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166702 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc378e8-c76a-42c2-b879-cd3b816f8268" containerName="manager" Jul 20 18:07:21.166798 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166714 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc378e8-c76a-42c2-b879-cd3b816f8268" containerName="manager" Jul 20 18:07:21.166798 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166733 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd6bdd4-e89e-4249-a047-9fd171492a61" containerName="authorino" Jul 20 18:07:21.166798 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166740 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd6bdd4-e89e-4249-a047-9fd171492a61" containerName="authorino" Jul 20 18:07:21.166928 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166801 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd6bdd4-e89e-4249-a047-9fd171492a61" containerName="authorino" Jul 20 18:07:21.166928 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.166808 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc378e8-c76a-42c2-b879-cd3b816f8268" containerName="manager" Jul 20 18:07:21.169707 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.169690 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.173532 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.173510 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 18:07:21.173532 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.173528 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 18:07:21.173754 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.173610 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mk6b9\"" Jul 20 18:07:21.179555 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.179533 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:21.253909 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.253879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.254062 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.253961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh6w\" (UniqueName: \"kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.355342 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.355314 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.355475 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.355403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh6w\" (UniqueName: \"kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.357824 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.357804 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.364173 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.364142 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh6w\" (UniqueName: \"kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w\") pod \"maas-controller-584c885fff-cs4cg\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.481793 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.481721 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:21.611033 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:21.610993 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:22.569233 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:22.569194 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-cs4cg" event={"ID":"ff99b3eb-722d-460b-a92b-c115ed286671","Type":"ContainerStarted","Data":"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8"} Jul 20 18:07:22.569784 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:22.569252 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-cs4cg" event={"ID":"ff99b3eb-722d-460b-a92b-c115ed286671","Type":"ContainerStarted","Data":"204b4cb64609ca2c62cf12739d884fd52793e20068ec4b1ff004da5b8ac0073f"} Jul 20 18:07:22.569784 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:22.569323 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:22.587188 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:22.587140 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-584c885fff-cs4cg" podStartSLOduration=0.863988368 podStartE2EDuration="1.587125483s" totalImagesPullingTime="723.137115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:07:22.585791342 +0000 UTC m=+1170.524069633" watchObservedRunningTime="2026-07-20 18:07:22.587125483 +0000 UTC m=+1170.525403774" Jul 20 18:07:33.576369 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:33.576336 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:38.240324 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.240292 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:38.240747 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.240500 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-584c885fff-cs4cg" podUID="ff99b3eb-722d-460b-a92b-c115ed286671" containerName="manager" containerID="cri-o://7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8" gracePeriod=30 Jul 20 18:07:38.493812 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.493754 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:38.604121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.604084 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnh6w\" (UniqueName: \"kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w\") pod \"ff99b3eb-722d-460b-a92b-c115ed286671\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " Jul 20 18:07:38.604286 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.604155 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert\") pod \"ff99b3eb-722d-460b-a92b-c115ed286671\" (UID: \"ff99b3eb-722d-460b-a92b-c115ed286671\") " Jul 20 18:07:38.606245 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.606217 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w" (OuterVolumeSpecName: "kube-api-access-xnh6w") pod "ff99b3eb-722d-460b-a92b-c115ed286671" (UID: "ff99b3eb-722d-460b-a92b-c115ed286671"). InnerVolumeSpecName "kube-api-access-xnh6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:07:38.606324 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.606279 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ff99b3eb-722d-460b-a92b-c115ed286671" (UID: "ff99b3eb-722d-460b-a92b-c115ed286671"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:07:38.640775 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.640743 2554 generic.go:356] "Generic (PLEG): container finished" podID="ff99b3eb-722d-460b-a92b-c115ed286671" containerID="7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8" exitCode=0 Jul 20 18:07:38.640899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.640777 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-cs4cg" event={"ID":"ff99b3eb-722d-460b-a92b-c115ed286671","Type":"ContainerDied","Data":"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8"} Jul 20 18:07:38.640899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.640805 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-cs4cg" Jul 20 18:07:38.640899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.640842 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-cs4cg" event={"ID":"ff99b3eb-722d-460b-a92b-c115ed286671","Type":"ContainerDied","Data":"204b4cb64609ca2c62cf12739d884fd52793e20068ec4b1ff004da5b8ac0073f"} Jul 20 18:07:38.640899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.640873 2554 scope.go:117] "RemoveContainer" containerID="7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8" Jul 20 18:07:38.651909 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.651831 2554 scope.go:117] "RemoveContainer" containerID="7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8" Jul 20 18:07:38.652481 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:07:38.652460 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8\": container with ID starting with 7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8 not found: ID does not exist" containerID="7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8" Jul 20 18:07:38.652593 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.652487 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8"} err="failed to get container status \"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8\": rpc error: code = NotFound desc = could not find container \"7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8\": container with ID starting with 7fe21b32fc0fd033cf8e25275af0ecc1288f1ad00586c372b6a862cdf4f0c2f8 not found: ID does not exist" Jul 20 18:07:38.667062 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.667041 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:38.668572 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.668553 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-584c885fff-cs4cg"] Jul 20 18:07:38.705265 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.705243 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff99b3eb-722d-460b-a92b-c115ed286671-webhook-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:07:38.705265 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:38.705263 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh6w\" (UniqueName: \"kubernetes.io/projected/ff99b3eb-722d-460b-a92b-c115ed286671-kube-api-access-xnh6w\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:07:39.475899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.475870 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-584c885fff-f52zw"] Jul 20 18:07:39.476263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.476213 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff99b3eb-722d-460b-a92b-c115ed286671" containerName="manager" Jul 20 18:07:39.476263 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.476224 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff99b3eb-722d-460b-a92b-c115ed286671" containerName="manager" Jul 20 18:07:39.476343 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.476317 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff99b3eb-722d-460b-a92b-c115ed286671" containerName="manager" Jul 20 18:07:39.480487 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.480470 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.484600 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.484580 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 20 18:07:39.484735 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.484580 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mk6b9\"" Jul 20 18:07:39.484735 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.484615 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 20 18:07:39.490600 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.490581 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-f52zw"] Jul 20 18:07:39.613416 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.613383 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzxj\" (UniqueName: \"kubernetes.io/projected/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-kube-api-access-2tzxj\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.613598 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.613433 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-webhook-cert\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.714693 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.714587 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-webhook-cert\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.714833 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.714801 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzxj\" (UniqueName: \"kubernetes.io/projected/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-kube-api-access-2tzxj\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.717370 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.717331 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-webhook-cert\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.724243 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.724214 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzxj\" (UniqueName: \"kubernetes.io/projected/0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089-kube-api-access-2tzxj\") pod \"maas-controller-584c885fff-f52zw\" (UID: \"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089\") " pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.791612 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.791581 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:39.923820 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:39.923795 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c885fff-f52zw"] Jul 20 18:07:39.925417 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:07:39.925389 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4e4c8a_8179_4c4f_b4d7_b1a490b5d089.slice/crio-a0aa793729c560fe514451b82be825de3c90867267a4df35e83df66c0d2a0f70 WatchSource:0}: Error finding container a0aa793729c560fe514451b82be825de3c90867267a4df35e83df66c0d2a0f70: Status 404 returned error can't find the container with id a0aa793729c560fe514451b82be825de3c90867267a4df35e83df66c0d2a0f70 Jul 20 18:07:40.550520 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:40.550481 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff99b3eb-722d-460b-a92b-c115ed286671" path="/var/lib/kubelet/pods/ff99b3eb-722d-460b-a92b-c115ed286671/volumes" Jul 20 18:07:40.652413 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:40.652323 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-f52zw" event={"ID":"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089","Type":"ContainerStarted","Data":"be16242b6dba94a108f4ac6800a8152b5b452b9c89a5607ba2729a15e1c95a18"} Jul 20 18:07:40.652413 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:40.652379 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c885fff-f52zw" event={"ID":"0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089","Type":"ContainerStarted","Data":"a0aa793729c560fe514451b82be825de3c90867267a4df35e83df66c0d2a0f70"} Jul 20 18:07:40.652659 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:40.652491 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:40.672472 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:40.672425 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-584c885fff-f52zw" podStartSLOduration=1.292171271 podStartE2EDuration="1.67240843s" totalImagesPullingTime="380.237159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:07:40.671068047 +0000 UTC m=+1188.609346351" watchObservedRunningTime="2026-07-20 18:07:40.67240843 +0000 UTC m=+1188.610686710" Jul 20 18:07:51.659546 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:51.659515 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-584c885fff-f52zw" Jul 20 18:07:52.495690 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:52.495661 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:07:52.496653 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:07:52.496614 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:12:52.524122 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:12:52.524091 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:12:52.525603 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:12:52.525580 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:15:56.514814 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.514777 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:15:56.518243 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.518224 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.521899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.521879 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b075c996-kserve-self-signed-certs\"" Jul 20 18:15:56.534738 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.534710 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:15:56.612537 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.612712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612549 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.612712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnmc\" (UniqueName: \"kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.612712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612677 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.612712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612706 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.612917 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.612730 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713319 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713282 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnmc\" (UniqueName: \"kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713319 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713322 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713547 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713353 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713547 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713673 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713692 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713770 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.713947 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.713928 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.714045 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.714021 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.715869 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.715848 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.716152 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.716131 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.722351 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.722326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnmc\" (UniqueName: \"kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc\") pod \"e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.830471 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.830370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:15:56.977491 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:56.977463 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:15:56.980985 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:15:56.980955 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6597a17_44ab_4011_a230_d19263271b8f.slice/crio-ff7c1d3b0268176b32e1a7eba8af10dd30db314cc6009afdfd79e4fffc7e699f WatchSource:0}: Error finding container ff7c1d3b0268176b32e1a7eba8af10dd30db314cc6009afdfd79e4fffc7e699f: Status 404 returned error can't find the container with id ff7c1d3b0268176b32e1a7eba8af10dd30db314cc6009afdfd79e4fffc7e699f Jul 20 18:15:57.978893 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:57.978861 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" event={"ID":"a6597a17-44ab-4011-a230-d19263271b8f","Type":"ContainerStarted","Data":"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc"} Jul 20 18:15:57.979275 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:57.978902 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" event={"ID":"a6597a17-44ab-4011-a230-d19263271b8f","Type":"ContainerStarted","Data":"ff7c1d3b0268176b32e1a7eba8af10dd30db314cc6009afdfd79e4fffc7e699f"} Jul 20 18:15:57.998052 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:15:57.998003 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" podStartSLOduration=1.997989199 podStartE2EDuration="1.997989199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:15:57.996954606 +0000 UTC m=+1685.935232900" watchObservedRunningTime="2026-07-20 18:15:57.997989199 +0000 UTC m=+1685.936267492" Jul 20 18:16:06.830619 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:06.830580 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:06.830619 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:06.830657 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:06.843329 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:06.843299 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:07.040520 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:07.040483 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:34.523997 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:34.523953 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:16:49.546489 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.546442 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" podUID="a6597a17-44ab-4011-a230-d19263271b8f" containerName="main" containerID="cri-o://539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc" gracePeriod=45 Jul 20 18:16:49.795270 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.795247 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:49.889984 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.889912 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.889984 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.889949 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.890195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.889988 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.890195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890004 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.890195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890031 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.890195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890072 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dnmc\" (UniqueName: \"kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc\") pod \"a6597a17-44ab-4011-a230-d19263271b8f\" (UID: \"a6597a17-44ab-4011-a230-d19263271b8f\") " Jul 20 18:16:49.890195 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890171 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:49.890424 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890264 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:49.890424 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890328 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:49.890424 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.890330 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home" (OuterVolumeSpecName: "home") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:49.892205 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.892171 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm" (OuterVolumeSpecName: "dshm") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:16:49.892299 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.892207 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:16:49.892299 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.892255 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc" (OuterVolumeSpecName: "kube-api-access-7dnmc") pod "a6597a17-44ab-4011-a230-d19263271b8f" (UID: "a6597a17-44ab-4011-a230-d19263271b8f"). InnerVolumeSpecName "kube-api-access-7dnmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:16:49.990788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.990761 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnmc\" (UniqueName: \"kubernetes.io/projected/a6597a17-44ab-4011-a230-d19263271b8f-kube-api-access-7dnmc\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:49.990788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.990786 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6597a17-44ab-4011-a230-d19263271b8f-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:49.990914 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.990795 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:49.990914 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.990802 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:49.990914 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:49.990810 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6597a17-44ab-4011-a230-d19263271b8f-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:16:50.229647 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.229550 2554 generic.go:356] "Generic (PLEG): container finished" podID="a6597a17-44ab-4011-a230-d19263271b8f" containerID="539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc" exitCode=0 Jul 20 18:16:50.229647 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.229594 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" Jul 20 18:16:50.229797 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.229652 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" event={"ID":"a6597a17-44ab-4011-a230-d19263271b8f","Type":"ContainerDied","Data":"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc"} Jul 20 18:16:50.229797 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.229699 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw" event={"ID":"a6597a17-44ab-4011-a230-d19263271b8f","Type":"ContainerDied","Data":"ff7c1d3b0268176b32e1a7eba8af10dd30db314cc6009afdfd79e4fffc7e699f"} Jul 20 18:16:50.229797 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.229716 2554 scope.go:117] "RemoveContainer" containerID="539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc" Jul 20 18:16:50.239131 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.239114 2554 scope.go:117] "RemoveContainer" containerID="539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc" Jul 20 18:16:50.239406 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:16:50.239389 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc\": container with ID starting with 539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc not found: ID does not exist" containerID="539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc" Jul 20 18:16:50.239460 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.239415 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc"} err="failed to get container status \"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc\": rpc error: code = NotFound desc = could not find container \"539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc\": container with ID starting with 539e30c89ae5afda78676ba5baea8d3fdd9b46b0a2de0a7021944d27a21548cc not found: ID does not exist" Jul 20 18:16:50.252193 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.252171 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:16:50.254025 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.254007 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b075c996-kserve-755dbb6779-hl4tw"] Jul 20 18:16:50.549552 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:16:50.549524 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6597a17-44ab-4011-a230-d19263271b8f" path="/var/lib/kubelet/pods/a6597a17-44ab-4011-a230-d19263271b8f/volumes" Jul 20 18:17:52.554538 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:17:52.554512 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:17:52.557399 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:17:52.557375 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:20:14.094753 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.094694 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:20:14.097242 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.095118 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6597a17-44ab-4011-a230-d19263271b8f" containerName="main" Jul 20 18:20:14.097242 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.095131 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6597a17-44ab-4011-a230-d19263271b8f" containerName="main" Jul 20 18:20:14.097242 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.095204 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6597a17-44ab-4011-a230-d19263271b8f" containerName="main" Jul 20 18:20:14.098176 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.098156 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.101968 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.101949 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-337dced0-openshift-default-dockercfg-cjqpc\"" Jul 20 18:20:14.124264 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.124229 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:20:14.182091 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182058 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182091 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182097 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182315 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182123 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182315 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182315 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182256 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182315 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182273 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182449 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182319 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182449 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182346 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsnm\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.182449 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.182399 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283008 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.282968 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283201 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283019 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283201 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283201 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283080 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283201 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283100 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283231 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsnm\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283364 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283676 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283503 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283676 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283534 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283787 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283735 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.283842 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.283798 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.284121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.284096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.285525 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.285504 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.285711 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.285695 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.293023 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.292999 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.293214 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.293192 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsnm\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm\") pod \"e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.408769 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.408671 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:14.556650 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.553583 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:20:14.559492 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.559450 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:14.559794 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.559767 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:14.559886 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:14.559823 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:15.216293 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:15.216257 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerStarted","Data":"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460"} Jul 20 18:20:15.216724 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:15.216310 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerStarted","Data":"b9220185082c403df374fb25275ca9ab1171784ba59da22079e5c9884c9394ff"} Jul 20 18:20:15.240478 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:15.240425 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podStartSLOduration=1.240410772 podStartE2EDuration="1.240410772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:20:15.238111436 +0000 UTC m=+1943.176389739" watchObservedRunningTime="2026-07-20 18:20:15.240410772 +0000 UTC m=+1943.178689112" Jul 20 18:20:15.409560 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:15.409526 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:15.414213 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:15.414189 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:16.222141 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:16.222112 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:16.223272 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:16.223252 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:31.224226 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.224190 2554 patch_prober.go:28] interesting pod/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 20 18:20:31.224699 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.224258 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 20 18:20:31.298267 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.298193 2554 generic.go:356] "Generic (PLEG): container finished" podID="9a415ce2-366e-4945-bde8-9373e43e0404" containerID="dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460" exitCode=0 Jul 20 18:20:31.298447 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.298271 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerDied","Data":"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460"} Jul 20 18:20:31.298694 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.298680 2554 scope.go:117] "RemoveContainer" containerID="dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460" Jul 20 18:20:31.299666 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.299641 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:31.299744 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.299710 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:31.299781 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:31.299756 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:20:32.304989 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:32.304952 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerStarted","Data":"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09"} Jul 20 18:20:32.409828 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:32.409791 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:32.410040 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:32.410020 2554 patch_prober.go:28] interesting pod/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 20 18:20:32.410104 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:32.410067 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 20 18:20:33.409973 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:33.409934 2554 patch_prober.go:28] interesting pod/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 20 18:20:33.410380 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:33.410001 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 20 18:20:34.409793 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:34.409755 2554 patch_prober.go:28] interesting pod/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 20 18:20:34.409981 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:34.409827 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 20 18:20:35.409484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:35.409442 2554 patch_prober.go:28] interesting pod/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 20 18:20:35.409946 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:35.409504 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 20 18:20:36.413504 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:36.413472 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:36.413912 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:36.413853 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:36.414566 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:36.414549 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:20:57.547960 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:57.547876 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:20:57.548495 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:20:57.548136 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" containerID="cri-o://3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09" gracePeriod=30 Jul 20 18:21:02.801927 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.801901 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:21:02.920323 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920221 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920323 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920273 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920353 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920407 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920431 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920462 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920483 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920565 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920546 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:02.920986 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920563 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qsnm\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920986 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920663 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert\") pod \"9a415ce2-366e-4945-bde8-9373e43e0404\" (UID: \"9a415ce2-366e-4945-bde8-9373e43e0404\") " Jul 20 18:21:02.920986 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.920786 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:02.921153 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.921044 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:02.921153 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.921047 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:02.921153 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.921064 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-workload-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:02.921153 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.921112 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:21:02.921367 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.921334 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:02.922754 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.922722 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:21:02.922885 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.922804 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm" (OuterVolumeSpecName: "kube-api-access-2qsnm") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "kube-api-access-2qsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:21:02.923002 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.922973 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:21:02.923061 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:02.923043 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a415ce2-366e-4945-bde8-9373e43e0404" (UID: "9a415ce2-366e-4945-bde8-9373e43e0404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:21:03.021481 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021449 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a415ce2-366e-4945-bde8-9373e43e0404-istiod-ca-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021481 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021478 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-envoy\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021481 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021490 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-credential-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021502 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a415ce2-366e-4945-bde8-9373e43e0404-istio-data\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021515 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a415ce2-366e-4945-bde8-9373e43e0404-istio-podinfo\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021526 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-istio-token\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.021770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.021537 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qsnm\" (UniqueName: \"kubernetes.io/projected/9a415ce2-366e-4945-bde8-9373e43e0404-kube-api-access-2qsnm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:21:03.466126 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.466091 2554 generic.go:356] "Generic (PLEG): container finished" podID="9a415ce2-366e-4945-bde8-9373e43e0404" containerID="3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09" exitCode=0 Jul 20 18:21:03.466340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.466151 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" Jul 20 18:21:03.466340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.466180 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerDied","Data":"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09"} Jul 20 18:21:03.466340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.466239 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6" event={"ID":"9a415ce2-366e-4945-bde8-9373e43e0404","Type":"ContainerDied","Data":"b9220185082c403df374fb25275ca9ab1171784ba59da22079e5c9884c9394ff"} Jul 20 18:21:03.466340 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.466262 2554 scope.go:117] "RemoveContainer" containerID="3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09" Jul 20 18:21:03.476009 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.475992 2554 scope.go:117] "RemoveContainer" containerID="dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460" Jul 20 18:21:03.484433 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.484415 2554 scope.go:117] "RemoveContainer" containerID="3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09" Jul 20 18:21:03.484720 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:21:03.484700 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09\": container with ID starting with 3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09 not found: ID does not exist" containerID="3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09" Jul 20 18:21:03.484804 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.484731 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09"} err="failed to get container status \"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09\": rpc error: code = NotFound desc = could not find container \"3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09\": container with ID starting with 3dddf6ee741579e399747be2f347bd63b06cf5fe0af2428b6683b598b1c21a09 not found: ID does not exist" Jul 20 18:21:03.484804 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.484758 2554 scope.go:117] "RemoveContainer" containerID="dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460" Jul 20 18:21:03.485013 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:21:03.484994 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460\": container with ID starting with dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460 not found: ID does not exist" containerID="dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460" Jul 20 18:21:03.485056 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.485021 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460"} err="failed to get container status \"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460\": rpc error: code = NotFound desc = could not find container \"dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460\": container with ID starting with dce2a36b36aaec04a1bb29186809ebad89bd2d9017c6bad4c41bffba56345460 not found: ID does not exist" Jul 20 18:21:03.489212 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.489189 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:21:03.495329 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:03.495305 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-337dced0-openshift-default-5988b89cc7-k9ln6"] Jul 20 18:21:04.551165 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:21:04.551131 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" path="/var/lib/kubelet/pods/9a415ce2-366e-4945-bde8-9373e43e0404/volumes" Jul 20 18:22:52.586675 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:22:52.586584 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:22:52.589663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:22:52.589621 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:25:57.488794 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.488756 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:25:57.491170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.489193 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:25:57.491170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.489206 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:25:57.491170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.489216 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:25:57.491170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.489223 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:25:57.491170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.489273 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:25:57.492243 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.492217 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.496078 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.496048 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2be07642\"/\"default-dockercfg-fq7w7\"" Jul 20 18:25:57.496826 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.496806 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2be07642\"/\"kube-root-ca.crt\"" Jul 20 18:25:57.497052 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.497036 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2be07642\"/\"e2e-lifecycle-model-2be07642-kserve-self-signed-certs\"" Jul 20 18:25:57.497134 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.497094 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2be07642\"/\"openshift-service-ca.crt\"" Jul 20 18:25:57.505192 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505166 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.505338 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505208 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.505338 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505242 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6gx\" (UniqueName: \"kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.505464 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505349 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.505464 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.505554 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.505460 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.506586 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.506565 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:25:57.606467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606427 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606522 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606547 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606712 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606569 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6gx\" (UniqueName: \"kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606947 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606770 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.606947 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.606913 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.607059 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.607022 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.607188 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.607168 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.608925 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.608902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.609121 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.609104 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.616592 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.616565 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6gx\" (UniqueName: \"kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx\") pod \"e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.802257 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.802219 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:25:57.948229 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:57.948205 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:25:57.950939 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:25:57.950910 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55e77e1_f420_4f2d_8b5f_30af20eeaa49.slice/crio-a6f5589a073447909e127b1b318c29339db4e29b7eff0b3f25113d09f677f62b WatchSource:0}: Error finding container a6f5589a073447909e127b1b318c29339db4e29b7eff0b3f25113d09f677f62b: Status 404 returned error can't find the container with id a6f5589a073447909e127b1b318c29339db4e29b7eff0b3f25113d09f677f62b Jul 20 18:25:58.856338 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:58.856304 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" event={"ID":"a55e77e1-f420-4f2d-8b5f-30af20eeaa49","Type":"ContainerStarted","Data":"5d76b87f16181dfcb89180daac0f57f3d8a7c8048ce4f751cd85804873579710"} Jul 20 18:25:58.856338 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:58.856347 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" event={"ID":"a55e77e1-f420-4f2d-8b5f-30af20eeaa49","Type":"ContainerStarted","Data":"a6f5589a073447909e127b1b318c29339db4e29b7eff0b3f25113d09f677f62b"} Jul 20 18:25:58.880442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:25:58.880388 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" podStartSLOduration=1.880372846 podStartE2EDuration="1.880372846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:25:58.879898865 +0000 UTC m=+2286.818177155" watchObservedRunningTime="2026-07-20 18:25:58.880372846 +0000 UTC m=+2286.818651172" Jul 20 18:26:07.802549 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:07.802494 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:26:07.803177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:07.802569 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:26:07.815179 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:07.815146 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:26:07.908644 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:07.908601 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:26:44.902308 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:44.902228 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:26:59.922602 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:26:59.922536 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" podUID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" containerName="main" containerID="cri-o://5d76b87f16181dfcb89180daac0f57f3d8a7c8048ce4f751cd85804873579710" gracePeriod=45 Jul 20 18:27:00.149433 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.149408 2554 generic.go:356] "Generic (PLEG): container finished" podID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" containerID="5d76b87f16181dfcb89180daac0f57f3d8a7c8048ce4f751cd85804873579710" exitCode=0 Jul 20 18:27:00.149522 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.149466 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" event={"ID":"a55e77e1-f420-4f2d-8b5f-30af20eeaa49","Type":"ContainerDied","Data":"5d76b87f16181dfcb89180daac0f57f3d8a7c8048ce4f751cd85804873579710"} Jul 20 18:27:00.175377 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.175311 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:27:00.248937 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.248900 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249134 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249008 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc6gx\" (UniqueName: \"kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249134 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249066 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249134 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249120 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249302 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249146 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249302 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249178 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs\") pod \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\" (UID: \"a55e77e1-f420-4f2d-8b5f-30af20eeaa49\") " Jul 20 18:27:00.249474 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249166 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:00.249474 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249355 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home" (OuterVolumeSpecName: "home") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:00.249474 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.249373 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache" (OuterVolumeSpecName: "model-cache") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:00.251313 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.251279 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:27:00.251425 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.251324 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm" (OuterVolumeSpecName: "dshm") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:27:00.251425 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.251391 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx" (OuterVolumeSpecName: "kube-api-access-jc6gx") pod "a55e77e1-f420-4f2d-8b5f-30af20eeaa49" (UID: "a55e77e1-f420-4f2d-8b5f-30af20eeaa49"). InnerVolumeSpecName "kube-api-access-jc6gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:27:00.350816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350781 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6gx\" (UniqueName: \"kubernetes.io/projected/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-kube-api-access-jc6gx\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:00.350816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350810 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:00.350816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350820 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:00.351043 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350828 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:00.351043 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350835 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:00.351043 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:00.350842 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55e77e1-f420-4f2d-8b5f-30af20eeaa49-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:27:01.154891 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:01.154856 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" event={"ID":"a55e77e1-f420-4f2d-8b5f-30af20eeaa49","Type":"ContainerDied","Data":"a6f5589a073447909e127b1b318c29339db4e29b7eff0b3f25113d09f677f62b"} Jul 20 18:27:01.155327 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:01.154903 2554 scope.go:117] "RemoveContainer" containerID="5d76b87f16181dfcb89180daac0f57f3d8a7c8048ce4f751cd85804873579710" Jul 20 18:27:01.155327 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:01.154864 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr" Jul 20 18:27:01.176023 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:01.175998 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:27:01.181972 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:01.181945 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2be07642/e2e-lifecycle-model-2be07642-kserve-f6667d8b7-xq7mr"] Jul 20 18:27:02.550103 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:02.550069 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" path="/var/lib/kubelet/pods/a55e77e1-f420-4f2d-8b5f-30af20eeaa49/volumes" Jul 20 18:27:52.613412 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:52.613381 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:27:52.617202 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:27:52.617177 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:29:01.626667 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.626619 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:29:01.627194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.627021 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" containerName="main" Jul 20 18:29:01.627194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.627034 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" containerName="main" Jul 20 18:29:01.627194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.627089 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a415ce2-366e-4945-bde8-9373e43e0404" containerName="istio-proxy" Jul 20 18:29:01.627194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.627100 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55e77e1-f420-4f2d-8b5f-30af20eeaa49" containerName="main" Jul 20 18:29:01.630173 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.630151 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.633427 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.633403 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e73607-openshift-default-dockercfg-cmr65\"" Jul 20 18:29:01.645149 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.645124 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:29:01.650060 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650040 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650168 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650076 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650219 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650189 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650266 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650234 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650309 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650274 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650357 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650399 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650356 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650452 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.650506 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.650463 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751228 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751178 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751304 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751355 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751418 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751467 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751809 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751501 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751809 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751535 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.751967 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.751939 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.752272 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.752245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.752540 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.752511 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.752540 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.752521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.753002 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.752975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.761234 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.761191 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.761718 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.761693 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.764055 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.764010 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.764216 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.764049 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token\") pod \"e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:01.940658 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:01.940549 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:02.094102 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.094071 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:29:02.097913 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:29:02.097885 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b70cae_5ff2_4556_bc7f_3f9a961b42e5.slice/crio-568fcc5e601cece5e25b2a0ebc478eb980185d2346b12fc60e78271a8cc421e5 WatchSource:0}: Error finding container 568fcc5e601cece5e25b2a0ebc478eb980185d2346b12fc60e78271a8cc421e5: Status 404 returned error can't find the container with id 568fcc5e601cece5e25b2a0ebc478eb980185d2346b12fc60e78271a8cc421e5 Jul 20 18:29:02.100420 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.100385 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:02.100526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.100507 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:02.100593 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.100555 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:02.732335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.732287 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerStarted","Data":"00cd65e4fe424d51801b351ebfc104f24ff8751d81868ea5360f7270506e147c"} Jul 20 18:29:02.732335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.732343 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerStarted","Data":"568fcc5e601cece5e25b2a0ebc478eb980185d2346b12fc60e78271a8cc421e5"} Jul 20 18:29:02.767078 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.767020 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podStartSLOduration=1.767001442 podStartE2EDuration="1.767001442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:29:02.766516787 +0000 UTC m=+2470.704795089" watchObservedRunningTime="2026-07-20 18:29:02.767001442 +0000 UTC m=+2470.705279733" Jul 20 18:29:02.940713 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:02.940680 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:03.946743 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:03.946713 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:04.743300 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:04.743264 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:04.744234 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:04.744213 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:15.802235 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.802148 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerID="00cd65e4fe424d51801b351ebfc104f24ff8751d81868ea5360f7270506e147c" exitCode=0 Jul 20 18:29:15.802235 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.802221 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerDied","Data":"00cd65e4fe424d51801b351ebfc104f24ff8751d81868ea5360f7270506e147c"} Jul 20 18:29:15.802719 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.802582 2554 scope.go:117] "RemoveContainer" containerID="00cd65e4fe424d51801b351ebfc104f24ff8751d81868ea5360f7270506e147c" Jul 20 18:29:15.803636 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.803601 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:15.803699 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.803684 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:15.803736 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:15.803725 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:29:16.809183 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:16.809146 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerStarted","Data":"af8075eb2e17259584e6a0c5e066793aa2e58a63d52bb5ce07afce9ee9d33feb"} Jul 20 18:29:16.941720 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:16.941683 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:16.943097 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:16.943057 2554 patch_prober.go:28] interesting pod/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 20 18:29:16.943254 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:16.943141 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 20 18:29:17.941705 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:17.941662 2554 patch_prober.go:28] interesting pod/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 20 18:29:17.942222 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:17.941729 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 20 18:29:18.941501 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:18.941457 2554 patch_prober.go:28] interesting pod/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 20 18:29:18.941829 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:18.941795 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 20 18:29:19.743504 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:19.743464 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:19.941390 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:19.941342 2554 patch_prober.go:28] interesting pod/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 20 18:29:19.941602 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:19.941416 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 20 18:29:20.945279 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:20.945244 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:20.946369 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:20.946348 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:29:48.294209 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.293501 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:29:48.297711 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.297692 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.300679 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.300651 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e73607\"/\"kube-root-ca.crt\"" Jul 20 18:29:48.301982 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.301963 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e73607\"/\"default-dockercfg-8rrt9\"" Jul 20 18:29:48.302115 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.301982 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e73607\"/\"openshift-service-ca.crt\"" Jul 20 18:29:48.302115 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.301986 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e73607\"/\"test-model-e73607-kserve-self-signed-certs\"" Jul 20 18:29:48.311587 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.311563 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:29:48.395503 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395474 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.395697 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395517 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.395697 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395564 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.395697 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395586 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.395697 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395659 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx46q\" (UniqueName: \"kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.395838 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.395739 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497186 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497156 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497191 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497212 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497228 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx46q\" (UniqueName: \"kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497288 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497380 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497697 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497621 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497848 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497806 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.497961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.497935 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.499897 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.499868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.500022 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.499998 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.508545 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.508520 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx46q\" (UniqueName: \"kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q\") pod \"test-model-e73607-kserve-5b69cd7dcc-qbj67\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.606843 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.606769 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:48.742923 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.742897 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:29:48.745259 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:29:48.745228 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e79b9dc_0689_46fe_8c70_042f81e3057e.slice/crio-1859e7c0fb935dc1eaecfef60a74b17c63060135c3ff1cd2ab220def26262c3e WatchSource:0}: Error finding container 1859e7c0fb935dc1eaecfef60a74b17c63060135c3ff1cd2ab220def26262c3e: Status 404 returned error can't find the container with id 1859e7c0fb935dc1eaecfef60a74b17c63060135c3ff1cd2ab220def26262c3e Jul 20 18:29:48.964538 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.964444 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" event={"ID":"4e79b9dc-0689-46fe-8c70-042f81e3057e","Type":"ContainerStarted","Data":"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5"} Jul 20 18:29:48.964538 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.964484 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" event={"ID":"4e79b9dc-0689-46fe-8c70-042f81e3057e","Type":"ContainerStarted","Data":"1859e7c0fb935dc1eaecfef60a74b17c63060135c3ff1cd2ab220def26262c3e"} Jul 20 18:29:48.985359 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:48.985310 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" podStartSLOduration=0.985295035 podStartE2EDuration="985.295035ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:29:48.983953198 +0000 UTC m=+2516.922231497" watchObservedRunningTime="2026-07-20 18:29:48.985295035 +0000 UTC m=+2516.923573326" Jul 20 18:29:58.606917 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:58.606883 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:58.607316 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:58.606937 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:58.619804 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:58.619780 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:29:59.026118 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:29:59.026085 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:30:30.714072 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:30.714037 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:30:45.737791 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:45.737743 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" podUID="4e79b9dc-0689-46fe-8c70-042f81e3057e" containerName="main" containerID="cri-o://405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5" gracePeriod=45 Jul 20 18:30:45.999366 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:45.999339 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:30:46.134134 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134103 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134146 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134171 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134207 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134259 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134314 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134289 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx46q\" (UniqueName: \"kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q\") pod \"4e79b9dc-0689-46fe-8c70-042f81e3057e\" (UID: \"4e79b9dc-0689-46fe-8c70-042f81e3057e\") " Jul 20 18:30:46.134579 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134461 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:46.134579 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134546 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:46.134579 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134564 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home" (OuterVolumeSpecName: "home") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:46.134788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134654 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.134788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134669 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.134788 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.134678 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.136597 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.136562 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q" (OuterVolumeSpecName: "kube-api-access-dx46q") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "kube-api-access-dx46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:30:46.136742 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.136618 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:30:46.136742 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.136672 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm" (OuterVolumeSpecName: "dshm") pod "4e79b9dc-0689-46fe-8c70-042f81e3057e" (UID: "4e79b9dc-0689-46fe-8c70-042f81e3057e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:30:46.235401 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.235361 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx46q\" (UniqueName: \"kubernetes.io/projected/4e79b9dc-0689-46fe-8c70-042f81e3057e-kube-api-access-dx46q\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.235401 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.235395 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79b9dc-0689-46fe-8c70-042f81e3057e-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.235401 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.235407 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79b9dc-0689-46fe-8c70-042f81e3057e-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:30:46.247172 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.247135 2554 generic.go:356] "Generic (PLEG): container finished" podID="4e79b9dc-0689-46fe-8c70-042f81e3057e" containerID="405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5" exitCode=0 Jul 20 18:30:46.247335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.247184 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" Jul 20 18:30:46.247335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.247218 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" event={"ID":"4e79b9dc-0689-46fe-8c70-042f81e3057e","Type":"ContainerDied","Data":"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5"} Jul 20 18:30:46.247335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.247274 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67" event={"ID":"4e79b9dc-0689-46fe-8c70-042f81e3057e","Type":"ContainerDied","Data":"1859e7c0fb935dc1eaecfef60a74b17c63060135c3ff1cd2ab220def26262c3e"} Jul 20 18:30:46.247335 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.247292 2554 scope.go:117] "RemoveContainer" containerID="405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5" Jul 20 18:30:46.256758 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.256730 2554 scope.go:117] "RemoveContainer" containerID="405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5" Jul 20 18:30:46.257033 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:30:46.257015 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5\": container with ID starting with 405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5 not found: ID does not exist" containerID="405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5" Jul 20 18:30:46.257088 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.257046 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5"} err="failed to get container status \"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5\": rpc error: code = NotFound desc = could not find container \"405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5\": container with ID starting with 405501ac6b6aac4c8b354569a6949c1eac39b322420843c6dd8a64158e0a08c5 not found: ID does not exist" Jul 20 18:30:46.270050 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.270018 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:30:46.272304 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.272281 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e73607/test-model-e73607-kserve-5b69cd7dcc-qbj67"] Jul 20 18:30:46.551915 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:30:46.551828 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e79b9dc-0689-46fe-8c70-042f81e3057e" path="/var/lib/kubelet/pods/4e79b9dc-0689-46fe-8c70-042f81e3057e/volumes" Jul 20 18:31:15.197296 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:15.197206 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:31:15.197846 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:15.197471 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" containerID="cri-o://af8075eb2e17259584e6a0c5e066793aa2e58a63d52bb5ce07afce9ee9d33feb" gracePeriod=30 Jul 20 18:31:20.422384 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.422346 2554 generic.go:356] "Generic (PLEG): container finished" podID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerID="af8075eb2e17259584e6a0c5e066793aa2e58a63d52bb5ce07afce9ee9d33feb" exitCode=0 Jul 20 18:31:20.422847 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.422406 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerDied","Data":"af8075eb2e17259584e6a0c5e066793aa2e58a63d52bb5ce07afce9ee9d33feb"} Jul 20 18:31:20.422847 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.422508 2554 scope.go:117] "RemoveContainer" containerID="00cd65e4fe424d51801b351ebfc104f24ff8751d81868ea5360f7270506e147c" Jul 20 18:31:20.461829 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.461758 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:31:20.547936 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.547911 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.547936 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.547938 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548203 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.547970 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548203 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.547989 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548203 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548008 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548203 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548152 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548203 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548189 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548223 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:31:20.548484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548241 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:31:20.548484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548232 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548296 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:31:20.548484 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548319 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token\") pod \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\" (UID: \"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5\") " Jul 20 18:31:20.548795 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548503 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:31:20.548795 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548658 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.548795 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548680 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-credential-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.548795 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548695 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istiod-ca-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.548795 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.548708 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-workload-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.549039 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.549006 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:31:20.550771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.550739 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:31:20.550890 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.550818 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6" (OuterVolumeSpecName: "kube-api-access-wkzz6") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "kube-api-access-wkzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:31:20.550890 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.550845 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:31:20.550972 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.550891 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" (UID: "b8b70cae-5ff2-4556-bc7f-3f9a961b42e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:31:20.649313 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.649270 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-podinfo\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.649313 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.649303 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-data\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.649313 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.649318 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzz6\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-kube-api-access-wkzz6\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.649651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.649328 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-envoy\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:20.649651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:20.649337 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5-istio-token\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:31:21.429200 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:21.429163 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" event={"ID":"b8b70cae-5ff2-4556-bc7f-3f9a961b42e5","Type":"ContainerDied","Data":"568fcc5e601cece5e25b2a0ebc478eb980185d2346b12fc60e78271a8cc421e5"} Jul 20 18:31:21.429675 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:21.429220 2554 scope.go:117] "RemoveContainer" containerID="af8075eb2e17259584e6a0c5e066793aa2e58a63d52bb5ce07afce9ee9d33feb" Jul 20 18:31:21.429675 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:21.429168 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc" Jul 20 18:31:21.454821 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:21.454773 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:31:21.461836 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:21.461809 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e73607-openshift-default-84d8d5cb99-zwrmc"] Jul 20 18:31:22.551702 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:31:22.551666 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" path="/var/lib/kubelet/pods/b8b70cae-5ff2-4556-bc7f-3f9a961b42e5/volumes" Jul 20 18:32:36.001251 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001212 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001617 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001644 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001657 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001663 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001673 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e79b9dc-0689-46fe-8c70-042f81e3057e" containerName="main" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001678 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e79b9dc-0689-46fe-8c70-042f81e3057e" containerName="main" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001743 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001752 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e79b9dc-0689-46fe-8c70-042f81e3057e" containerName="main" Jul 20 18:32:36.001771 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.001757 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b70cae-5ff2-4556-bc7f-3f9a961b42e5" containerName="istio-proxy" Jul 20 18:32:36.004857 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.004835 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.007438 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.007420 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-daa21b-openshift-default-dockercfg-s65wv\"" Jul 20 18:32:36.018170 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.018143 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:32:36.031252 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031225 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031395 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031395 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031395 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031364 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031395 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031391 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031548 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031422 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031548 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031548 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031491 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.031673 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.031564 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pvl\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132193 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132193 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132198 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132239 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132300 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132336 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132367 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132395 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132786 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132449 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pvl\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132786 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132578 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132892 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132821 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.132943 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.132911 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.133037 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.133015 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.133231 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.133084 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.134934 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.134904 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.135165 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.135143 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.144457 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.144435 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.144601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.144582 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pvl\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl\") pod \"e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.314985 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.314903 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:36.467268 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.467239 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:32:36.469133 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:32:36.469105 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf297bf_2a97_4478_8d6b_c6e22076a934.slice/crio-8041821fa266175804bd9f78de24cc49cc05de40d35713983abf0ad29233629a WatchSource:0}: Error finding container 8041821fa266175804bd9f78de24cc49cc05de40d35713983abf0ad29233629a: Status 404 returned error can't find the container with id 8041821fa266175804bd9f78de24cc49cc05de40d35713983abf0ad29233629a Jul 20 18:32:36.471274 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.471242 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:36.471373 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.471332 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:36.471442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.471383 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:36.786468 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.786433 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerStarted","Data":"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9"} Jul 20 18:32:36.786694 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.786486 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerStarted","Data":"8041821fa266175804bd9f78de24cc49cc05de40d35713983abf0ad29233629a"} Jul 20 18:32:36.813341 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:36.813279 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podStartSLOduration=1.813260168 podStartE2EDuration="1.813260168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:32:36.812270631 +0000 UTC m=+2684.750548914" watchObservedRunningTime="2026-07-20 18:32:36.813260168 +0000 UTC m=+2684.751538459" Jul 20 18:32:37.315285 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:37.315249 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:37.320276 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:37.320248 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:37.791563 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:37.791532 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:37.792913 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:37.792889 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:52.640761 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.640659 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:32:52.646000 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.645973 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:32:52.793347 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.793310 2554 patch_prober.go:28] interesting pod/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 20 18:32:52.793511 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.793376 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 20 18:32:52.881895 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.881858 2554 generic.go:356] "Generic (PLEG): container finished" podID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerID="2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9" exitCode=0 Jul 20 18:32:52.882080 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.881929 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerDied","Data":"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9"} Jul 20 18:32:52.882365 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.882349 2554 scope.go:117] "RemoveContainer" containerID="2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9" Jul 20 18:32:52.883389 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.883357 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:52.883500 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.883426 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:52.883500 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:52.883465 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 18:32:53.888906 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:53.888869 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerStarted","Data":"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb"} Jul 20 18:32:54.315948 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:54.315913 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:54.316166 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:54.316090 2554 patch_prober.go:28] interesting pod/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 20 18:32:54.316166 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:54.316136 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 20 18:32:55.316587 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:55.316550 2554 patch_prober.go:28] interesting pod/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 20 18:32:55.317026 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:55.316649 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 20 18:32:56.316315 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:56.316280 2554 patch_prober.go:28] interesting pod/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 20 18:32:56.316521 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:56.316336 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 20 18:32:57.316046 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:57.316005 2554 patch_prober.go:28] interesting pod/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 20 18:32:57.316550 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:57.316073 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 20 18:32:58.319856 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:58.319813 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:58.320251 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:58.320096 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:32:58.320741 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:32:58.320723 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:33:42.798328 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.798296 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:33:42.801818 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.801800 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.805857 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.805828 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"default-dockercfg-bl5c2\"" Jul 20 18:33:42.805999 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.805872 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"kube-root-ca.crt\"" Jul 20 18:33:42.806050 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.806024 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"auth-test-model-d56278-kserve-self-signed-certs\"" Jul 20 18:33:42.806204 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.806184 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"openshift-service-ca.crt\"" Jul 20 18:33:42.814412 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.814388 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:33:42.960027 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.959987 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.960027 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.960026 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.960257 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.960052 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.960257 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.960133 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.960257 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.960210 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wkv\" (UniqueName: \"kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:42.960359 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:42.960264 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.061764 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061653 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.061764 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.061764 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061743 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061778 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061815 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.061874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wkv\" (UniqueName: \"kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062209 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.062184 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062278 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.062222 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.062278 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.062241 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.064002 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.063972 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.064262 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.064245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.071064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.071029 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wkv\" (UniqueName: \"kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv\") pod \"auth-test-model-d56278-kserve-6d77b76999-bkxsq\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.111105 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.111052 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:43.271406 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:43.271378 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:33:43.273231 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:33:43.273201 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad96b310_4146_410c_8dd0_dece7c4293ed.slice/crio-58a429edc68a3f8a7cae8fcec752a25fef45dd32db3816d1a5dd7b1a1e5c1a8a WatchSource:0}: Error finding container 58a429edc68a3f8a7cae8fcec752a25fef45dd32db3816d1a5dd7b1a1e5c1a8a: Status 404 returned error can't find the container with id 58a429edc68a3f8a7cae8fcec752a25fef45dd32db3816d1a5dd7b1a1e5c1a8a Jul 20 18:33:44.128645 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:44.128576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" event={"ID":"ad96b310-4146-410c-8dd0-dece7c4293ed","Type":"ContainerStarted","Data":"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a"} Jul 20 18:33:44.129030 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:44.128658 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" event={"ID":"ad96b310-4146-410c-8dd0-dece7c4293ed","Type":"ContainerStarted","Data":"58a429edc68a3f8a7cae8fcec752a25fef45dd32db3816d1a5dd7b1a1e5c1a8a"} Jul 20 18:33:44.163968 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:44.163916 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" podStartSLOduration=2.163899045 podStartE2EDuration="2.163899045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:33:44.161989229 +0000 UTC m=+2752.100267520" watchObservedRunningTime="2026-07-20 18:33:44.163899045 +0000 UTC m=+2752.102177335" Jul 20 18:33:53.111655 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:53.111595 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:53.111655 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:53.111667 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:53.124462 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:53.124432 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:33:53.182583 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:33:53.182554 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:35:47.697917 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:35:47.697818 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:36:02.718473 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:02.718413 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" podUID="ad96b310-4146-410c-8dd0-dece7c4293ed" containerName="main" containerID="cri-o://a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a" gracePeriod=45 Jul 20 18:36:02.969904 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:02.969833 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:36:03.166996 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.166966 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167038 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167062 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167090 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167177 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167167 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167380 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167189 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9wkv\" (UniqueName: \"kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv\") pod \"ad96b310-4146-410c-8dd0-dece7c4293ed\" (UID: \"ad96b310-4146-410c-8dd0-dece7c4293ed\") " Jul 20 18:36:03.167380 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167338 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:03.167380 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167352 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home" (OuterVolumeSpecName: "home") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:03.167535 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167480 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:03.167588 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167560 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.167588 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.167580 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.169410 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.169378 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm" (OuterVolumeSpecName: "dshm") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:36:03.169519 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.169428 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv" (OuterVolumeSpecName: "kube-api-access-f9wkv") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "kube-api-access-f9wkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:36:03.169519 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.169445 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad96b310-4146-410c-8dd0-dece7c4293ed" (UID: "ad96b310-4146-410c-8dd0-dece7c4293ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:36:03.268735 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.268704 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.268735 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.268736 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9wkv\" (UniqueName: \"kubernetes.io/projected/ad96b310-4146-410c-8dd0-dece7c4293ed-kube-api-access-f9wkv\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.268936 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.268747 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad96b310-4146-410c-8dd0-dece7c4293ed-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.268936 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.268755 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad96b310-4146-410c-8dd0-dece7c4293ed-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:36:03.818270 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.818232 2554 generic.go:356] "Generic (PLEG): container finished" podID="ad96b310-4146-410c-8dd0-dece7c4293ed" containerID="a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a" exitCode=0 Jul 20 18:36:03.818882 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.818293 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" Jul 20 18:36:03.818882 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.818337 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" event={"ID":"ad96b310-4146-410c-8dd0-dece7c4293ed","Type":"ContainerDied","Data":"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a"} Jul 20 18:36:03.818882 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.818377 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq" event={"ID":"ad96b310-4146-410c-8dd0-dece7c4293ed","Type":"ContainerDied","Data":"58a429edc68a3f8a7cae8fcec752a25fef45dd32db3816d1a5dd7b1a1e5c1a8a"} Jul 20 18:36:03.818882 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.818425 2554 scope.go:117] "RemoveContainer" containerID="a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a" Jul 20 18:36:03.830816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.830794 2554 scope.go:117] "RemoveContainer" containerID="a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a" Jul 20 18:36:03.831139 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:36:03.831111 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a\": container with ID starting with a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a not found: ID does not exist" containerID="a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a" Jul 20 18:36:03.831247 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.831149 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a"} err="failed to get container status \"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a\": rpc error: code = NotFound desc = could not find container \"a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a\": container with ID starting with a20c53d66ddf2c6660fc56bc379db8dbb8c36ed2e1b0c62796d02621909e004a not found: ID does not exist" Jul 20 18:36:03.842352 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.842320 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:36:03.846054 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:03.846032 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/auth-test-model-d56278-kserve-6d77b76999-bkxsq"] Jul 20 18:36:04.551046 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:04.551013 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad96b310-4146-410c-8dd0-dece7c4293ed" path="/var/lib/kubelet/pods/ad96b310-4146-410c-8dd0-dece7c4293ed/volumes" Jul 20 18:36:20.541095 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.541061 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:36:20.541680 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.541497 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad96b310-4146-410c-8dd0-dece7c4293ed" containerName="main" Jul 20 18:36:20.541680 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.541512 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad96b310-4146-410c-8dd0-dece7c4293ed" containerName="main" Jul 20 18:36:20.541680 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.541599 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad96b310-4146-410c-8dd0-dece7c4293ed" containerName="main" Jul 20 18:36:20.546236 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.546201 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.549979 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.549953 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"sub-test-model-d56278-kserve-self-signed-certs\"" Jul 20 18:36:20.550756 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.550729 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"kube-root-ca.crt\"" Jul 20 18:36:20.550883 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.550780 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"openshift-service-ca.crt\"" Jul 20 18:36:20.551131 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.551106 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"default-dockercfg-bl5c2\"" Jul 20 18:36:20.563554 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.563531 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:36:20.626324 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626285 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.626521 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.626521 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626406 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.626521 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626457 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbjr\" (UniqueName: \"kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.626708 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626526 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.626708 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.626573 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727287 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727252 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727287 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727287 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727580 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727580 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbjr\" (UniqueName: \"kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727580 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727778 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727755 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.727978 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.727948 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.728164 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.728084 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.728276 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.728215 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.730312 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.730287 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.730462 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.730447 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.739321 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.738940 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbjr\" (UniqueName: \"kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr\") pod \"sub-test-model-d56278-kserve-8567585579-5rg5c\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.858186 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.858095 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:20.995415 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:20.995389 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:36:20.997416 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:36:20.997382 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667d5715_71d5_4d82_ae6d_ccc3519f34ed.slice/crio-63e057cb40d7f33b0c31255a29625fd7fa2e2dcb6b382b417d6bfd75387ce412 WatchSource:0}: Error finding container 63e057cb40d7f33b0c31255a29625fd7fa2e2dcb6b382b417d6bfd75387ce412: Status 404 returned error can't find the container with id 63e057cb40d7f33b0c31255a29625fd7fa2e2dcb6b382b417d6bfd75387ce412 Jul 20 18:36:21.911502 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:21.911457 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" event={"ID":"667d5715-71d5-4d82-ae6d-ccc3519f34ed","Type":"ContainerStarted","Data":"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45"} Jul 20 18:36:21.911502 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:21.911500 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" event={"ID":"667d5715-71d5-4d82-ae6d-ccc3519f34ed","Type":"ContainerStarted","Data":"63e057cb40d7f33b0c31255a29625fd7fa2e2dcb6b382b417d6bfd75387ce412"} Jul 20 18:36:21.931529 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:21.931481 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" podStartSLOduration=1.931465091 podStartE2EDuration="1.931465091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:36:21.930372935 +0000 UTC m=+2909.868651273" watchObservedRunningTime="2026-07-20 18:36:21.931465091 +0000 UTC m=+2909.869743381" Jul 20 18:36:30.858252 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:30.858200 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:30.858696 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:30.858272 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:30.871414 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:30.871373 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:36:30.967874 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:36:30.967846 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:37:08.265451 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:08.265412 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:37:23.286211 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.286156 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" podUID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" containerName="main" containerID="cri-o://35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45" gracePeriod=45 Jul 20 18:37:23.558898 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.558870 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:37:23.626108 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626066 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.626406 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626136 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.626406 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626159 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.626406 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626184 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmbjr\" (UniqueName: \"kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.626596 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626378 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home" (OuterVolumeSpecName: "home") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:23.626596 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626424 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:23.626596 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626548 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:23.626596 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.626564 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:23.628414 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.628380 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:37:23.628539 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.628439 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr" (OuterVolumeSpecName: "kube-api-access-lmbjr") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "kube-api-access-lmbjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:37:23.727006 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.726969 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.727006 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.727009 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache\") pod \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\" (UID: \"667d5715-71d5-4d82-ae6d-ccc3519f34ed\") " Jul 20 18:37:23.727302 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.727285 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbjr\" (UniqueName: \"kubernetes.io/projected/667d5715-71d5-4d82-ae6d-ccc3519f34ed-kube-api-access-lmbjr\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:23.727358 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.727310 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667d5715-71d5-4d82-ae6d-ccc3519f34ed-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:23.727358 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.727336 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:23.729336 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.729309 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm" (OuterVolumeSpecName: "dshm") pod "667d5715-71d5-4d82-ae6d-ccc3519f34ed" (UID: "667d5715-71d5-4d82-ae6d-ccc3519f34ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:37:23.828344 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.828245 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:23.828344 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:23.828274 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667d5715-71d5-4d82-ae6d-ccc3519f34ed-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:37:24.214816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.214718 2554 generic.go:356] "Generic (PLEG): container finished" podID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" containerID="35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45" exitCode=0 Jul 20 18:37:24.214816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.214765 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" Jul 20 18:37:24.214816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.214807 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" event={"ID":"667d5715-71d5-4d82-ae6d-ccc3519f34ed","Type":"ContainerDied","Data":"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45"} Jul 20 18:37:24.215064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.214855 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c" event={"ID":"667d5715-71d5-4d82-ae6d-ccc3519f34ed","Type":"ContainerDied","Data":"63e057cb40d7f33b0c31255a29625fd7fa2e2dcb6b382b417d6bfd75387ce412"} Jul 20 18:37:24.215064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.214873 2554 scope.go:117] "RemoveContainer" containerID="35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45" Jul 20 18:37:24.225230 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.225209 2554 scope.go:117] "RemoveContainer" containerID="35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45" Jul 20 18:37:24.225482 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:37:24.225461 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45\": container with ID starting with 35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45 not found: ID does not exist" containerID="35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45" Jul 20 18:37:24.225535 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.225491 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45"} err="failed to get container status \"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45\": rpc error: code = NotFound desc = could not find container \"35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45\": container with ID starting with 35198ec20672ea5f86bea4ed009b7c1a978db2944a530e4946a8bd22250e5d45 not found: ID does not exist" Jul 20 18:37:24.238860 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.238829 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:37:24.242171 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.242142 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/sub-test-model-d56278-kserve-8567585579-5rg5c"] Jul 20 18:37:24.551093 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:24.551060 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" path="/var/lib/kubelet/pods/667d5715-71d5-4d82-ae6d-ccc3519f34ed/volumes" Jul 20 18:37:52.668839 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:52.668812 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:37:52.678518 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:52.678494 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k7dh8_c9e470a3-89d3-4902-9441-1b742929dbed/ovn-acl-logging/0.log" Jul 20 18:37:57.994903 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:57.994863 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:37:57.995651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:57.995606 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" containerName="main" Jul 20 18:37:57.995651 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:57.995647 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" containerName="main" Jul 20 18:37:57.995830 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:57.995759 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="667d5715-71d5-4d82-ae6d-ccc3519f34ed" containerName="main" Jul 20 18:37:57.999938 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:57.999917 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.002899 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.002878 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"openshift-service-ca.crt\"" Jul 20 18:37:58.004053 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.004029 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"rate-test-model-d56278-kserve-self-signed-certs\"" Jul 20 18:37:58.004140 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.004105 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"default-dockercfg-bl5c2\"" Jul 20 18:37:58.004200 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.004143 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d56278\"/\"kube-root-ca.crt\"" Jul 20 18:37:58.009564 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.009542 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:37:58.142108 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fknw\" (UniqueName: \"kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.142322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142124 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.142322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142143 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.142322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142191 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.142322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142225 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.142322 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.142249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243165 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243130 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fknw\" (UniqueName: \"kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243368 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243368 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243311 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243368 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243368 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243586 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243391 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243586 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243536 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243722 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243703 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.243814 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.243796 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.245682 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.245613 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.245903 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.245888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.252392 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.252372 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fknw\" (UniqueName: \"kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw\") pod \"rate-test-model-d56278-kserve-5d69499d7-clfs9\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.310610 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.310573 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:37:58.458461 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:58.458434 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:37:58.460620 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:37:58.460580 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea42ec01_7533_40df_bdba_7a9a994bc9b8.slice/crio-62e16457072f40e99f8e35e940a86f7f49c57cfda74b790a5206bebfa00fe54f WatchSource:0}: Error finding container 62e16457072f40e99f8e35e940a86f7f49c57cfda74b790a5206bebfa00fe54f: Status 404 returned error can't find the container with id 62e16457072f40e99f8e35e940a86f7f49c57cfda74b790a5206bebfa00fe54f Jul 20 18:37:59.398517 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:59.398480 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" event={"ID":"ea42ec01-7533-40df-bdba-7a9a994bc9b8","Type":"ContainerStarted","Data":"4cbb3a9d4417aa55e9c387c2b6ac5c11dd7538441f59e458d4ba9fd2d73850c3"} Jul 20 18:37:59.398517 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:59.398522 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" event={"ID":"ea42ec01-7533-40df-bdba-7a9a994bc9b8","Type":"ContainerStarted","Data":"62e16457072f40e99f8e35e940a86f7f49c57cfda74b790a5206bebfa00fe54f"} Jul 20 18:37:59.418482 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:37:59.418430 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" podStartSLOduration=2.418414127 podStartE2EDuration="2.418414127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:37:59.417305556 +0000 UTC m=+3007.355583845" watchObservedRunningTime="2026-07-20 18:37:59.418414127 +0000 UTC m=+3007.356692418" Jul 20 18:38:08.311461 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:38:08.311419 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:38:08.311898 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:38:08.311474 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:38:08.324181 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:38:08.324151 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:38:08.452829 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:38:08.452800 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:38:47.545765 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:38:47.545680 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:39:02.566270 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.566199 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" podUID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" containerName="main" containerID="cri-o://4cbb3a9d4417aa55e9c387c2b6ac5c11dd7538441f59e458d4ba9fd2d73850c3" gracePeriod=45 Jul 20 18:39:02.719113 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.719077 2554 generic.go:356] "Generic (PLEG): container finished" podID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" containerID="4cbb3a9d4417aa55e9c387c2b6ac5c11dd7538441f59e458d4ba9fd2d73850c3" exitCode=0 Jul 20 18:39:02.719308 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.719160 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" event={"ID":"ea42ec01-7533-40df-bdba-7a9a994bc9b8","Type":"ContainerDied","Data":"4cbb3a9d4417aa55e9c387c2b6ac5c11dd7538441f59e458d4ba9fd2d73850c3"} Jul 20 18:39:02.817998 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.817928 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:39:02.831803 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831775 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.831961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831808 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.831961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831827 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.831961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831859 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fknw\" (UniqueName: \"kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.831961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831889 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.831961 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.831946 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home\") pod \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\" (UID: \"ea42ec01-7533-40df-bdba-7a9a994bc9b8\") " Jul 20 18:39:02.832233 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.832058 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.832233 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.832211 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tmp-dir\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.832328 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.832219 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.832370 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.832333 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home" (OuterVolumeSpecName: "home") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.834077 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.834050 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm" (OuterVolumeSpecName: "dshm") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:02.834197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.834053 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 18:39:02.834197 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.834104 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw" (OuterVolumeSpecName: "kube-api-access-8fknw") pod "ea42ec01-7533-40df-bdba-7a9a994bc9b8" (UID: "ea42ec01-7533-40df-bdba-7a9a994bc9b8"). InnerVolumeSpecName "kube-api-access-8fknw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:02.932902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.932865 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-home\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.932902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.932897 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea42ec01-7533-40df-bdba-7a9a994bc9b8-tls-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.932902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.932908 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-dshm\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.933154 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.932918 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fknw\" (UniqueName: \"kubernetes.io/projected/ea42ec01-7533-40df-bdba-7a9a994bc9b8-kube-api-access-8fknw\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:02.933154 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:02.932927 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea42ec01-7533-40df-bdba-7a9a994bc9b8-model-cache\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:03.725284 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:03.725256 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" event={"ID":"ea42ec01-7533-40df-bdba-7a9a994bc9b8","Type":"ContainerDied","Data":"62e16457072f40e99f8e35e940a86f7f49c57cfda74b790a5206bebfa00fe54f"} Jul 20 18:39:03.725284 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:03.725275 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9" Jul 20 18:39:03.725860 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:03.725303 2554 scope.go:117] "RemoveContainer" containerID="4cbb3a9d4417aa55e9c387c2b6ac5c11dd7538441f59e458d4ba9fd2d73850c3" Jul 20 18:39:03.751700 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:03.751675 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:39:03.756431 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:03.756406 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d56278/rate-test-model-d56278-kserve-5d69499d7-clfs9"] Jul 20 18:39:04.550515 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:04.550469 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" path="/var/lib/kubelet/pods/ea42ec01-7533-40df-bdba-7a9a994bc9b8/volumes" Jul 20 18:39:28.578569 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:28.578535 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:39:28.579146 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:28.578772 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" containerID="cri-o://54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb" gracePeriod=30 Jul 20 18:39:33.843126 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.843058 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:39:33.878463 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.878432 2554 generic.go:356] "Generic (PLEG): container finished" podID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerID="54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb" exitCode=0 Jul 20 18:39:33.878663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.878492 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" Jul 20 18:39:33.878663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.878516 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerDied","Data":"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb"} Jul 20 18:39:33.878663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.878572 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8" event={"ID":"ddf297bf-2a97-4478-8d6b-c6e22076a934","Type":"ContainerDied","Data":"8041821fa266175804bd9f78de24cc49cc05de40d35713983abf0ad29233629a"} Jul 20 18:39:33.878663 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.878594 2554 scope.go:117] "RemoveContainer" containerID="54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb" Jul 20 18:39:33.887936 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.887916 2554 scope.go:117] "RemoveContainer" containerID="2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9" Jul 20 18:39:33.895770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.895750 2554 scope.go:117] "RemoveContainer" containerID="54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb" Jul 20 18:39:33.896023 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:39:33.896005 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb\": container with ID starting with 54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb not found: ID does not exist" containerID="54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb" Jul 20 18:39:33.896073 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.896031 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb"} err="failed to get container status \"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb\": rpc error: code = NotFound desc = could not find container \"54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb\": container with ID starting with 54f95968bedf2c1645e569185e8429278777b442679bae803497cf0820b680bb not found: ID does not exist" Jul 20 18:39:33.896073 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.896048 2554 scope.go:117] "RemoveContainer" containerID="2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9" Jul 20 18:39:33.896298 ip-10-0-135-108 kubenswrapper[2554]: E0720 18:39:33.896282 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9\": container with ID starting with 2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9 not found: ID does not exist" containerID="2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9" Jul 20 18:39:33.896387 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.896301 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9"} err="failed to get container status \"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9\": rpc error: code = NotFound desc = could not find container \"2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9\": container with ID starting with 2ca734a86489a4365cb3c6653a67cc58d03fe4f1650f93c7e0f93908f480bba9 not found: ID does not exist" Jul 20 18:39:33.923621 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923590 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923669 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923703 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923770 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923763 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923781 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923805 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89pvl\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923852 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.923902 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923882 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.924104 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923921 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:33.924104 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923934 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:33.924104 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.923982 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo\") pod \"ddf297bf-2a97-4478-8d6b-c6e22076a934\" (UID: \"ddf297bf-2a97-4478-8d6b-c6e22076a934\") " Jul 20 18:39:33.924254 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924122 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:33.924304 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924267 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 18:39:33.924398 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924362 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:33.924398 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924392 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf297bf-2a97-4478-8d6b-c6e22076a934-istiod-ca-cert\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:33.924611 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924406 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-credential-socket\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:33.924611 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.924416 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-workload-certs\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:33.925085 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.925058 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:33.926119 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.926090 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 18:39:33.926210 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.926144 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl" (OuterVolumeSpecName: "kube-api-access-89pvl") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "kube-api-access-89pvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:33.926395 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.926371 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 18:39:33.926478 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:33.926377 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddf297bf-2a97-4478-8d6b-c6e22076a934" (UID: "ddf297bf-2a97-4478-8d6b-c6e22076a934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 18:39:34.025562 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.025528 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-data\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:34.025562 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.025556 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pvl\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-kube-api-access-89pvl\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:34.025562 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.025567 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-token\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:34.025821 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.025576 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-podinfo\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:34.025821 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.025599 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf297bf-2a97-4478-8d6b-c6e22076a934-istio-envoy\") on node \"ip-10-0-135-108.ec2.internal\" DevicePath \"\"" Jul 20 18:39:34.202150 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.202116 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:39:34.206059 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.206033 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-daa21b-openshift-default-5579cfc9cf-dgfv8"] Jul 20 18:39:34.550597 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:34.550556 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" path="/var/lib/kubelet/pods/ddf297bf-2a97-4478-8d6b-c6e22076a934/volumes" Jul 20 18:39:52.263601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:52.263572 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b67867474-g8b4g_564a169a-4c89-4367-82f1-788b5cd2fa75/authorino/0.log" Jul 20 18:39:58.580919 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:58.580888 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cthmn_818c0036-9703-485a-babb-631ceeb82bfb/manager/0.log" Jul 20 18:39:58.717857 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:58.717827 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lh76v_dd06f2ad-09ea-4eb4-adaf-cee1749697ec/manager/0.log" Jul 20 18:39:58.870266 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:58.870176 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-584c885fff-f52zw_0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089/manager/0.log" Jul 20 18:39:59.004022 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:59.003995 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qp89n_df491736-5bfe-4358-9538-7a3adb0fd84b/server/0.log" Jul 20 18:39:59.140688 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:59.140601 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-75cmz_d032002b-b768-47e1-b32b-4c42377eede2/manager/0.log" Jul 20 18:39:59.447605 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:59.447519 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-mnlgv_f753c167-f037-420a-a79b-1843f6f5adf2/manager/0.log" Jul 20 18:39:59.454760 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:39:59.454737 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-mnlgv_f753c167-f037-420a-a79b-1843f6f5adf2/copy-manifests/0.log" Jul 20 18:40:00.340550 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:00.340521 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-48qpj_623215b7-9eae-4038-8a3c-7f6473161759/postgres/0.log" Jul 20 18:40:01.836581 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:01.836553 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b67867474-g8b4g_564a169a-4c89-4367-82f1-788b5cd2fa75/authorino/0.log" Jul 20 18:40:01.972091 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:01.972062 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z2ptg_b40d9a41-4fcc-48bb-ae51-6acad5a9034d/manager/0.log" Jul 20 18:40:02.108679 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:02.108582 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vq562_f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680/manager/0.log" Jul 20 18:40:02.365780 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:02.365702 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4vs7j_79e7c853-db8a-4810-a9a8-87d588931153/registry-server/0.log" Jul 20 18:40:02.797284 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:02.797254 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pn7sf_dab3ab50-e988-4b19-90fb-2506ed0768f8/manager/0.log" Jul 20 18:40:03.239816 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:03.239737 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45wngf_e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9/istio-proxy/0.log" Jul 20 18:40:03.622153 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:03.622123 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b99948476-qr9bh_8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c/kube-auth-proxy/0.log" Jul 20 18:40:03.757916 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:03.757885 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/1.log" Jul 20 18:40:03.883442 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:03.883360 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-w9mr7_3f02a0cf-de11-4252-bc3f-30720bb552aa/payload-pre-processing/0.log" Jul 20 18:40:04.015234 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:04.015205 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-nv7tm_cb9af6ee-cf84-4ab1-91d8-89547006f1ba/payload-processing/0.log" Jul 20 18:40:04.549683 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:04.549652 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z6xnh_21518261-e875-4fb8-b46c-84da91443053/main/0.log" Jul 20 18:40:04.952801 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:04.952713 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6bhnw_e270b02f-1cb6-4f79-9035-8dccbe789740/main/0.log" Jul 20 18:40:05.087084 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:05.087048 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2qmzr_928cfd54-ba59-40be-8c1c-2a3c614dee81/main/0.log" Jul 20 18:40:14.750400 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:14.750320 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mtfkc_48680f07-ab02-4e7f-8416-949da11961ba/global-pull-secret-syncer/0.log" Jul 20 18:40:14.914058 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:14.914026 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t8r5c_4fe3b281-c05a-47a1-80c2-696c417b0a48/konnectivity-agent/0.log" Jul 20 18:40:14.982369 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:14.982330 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-108.ec2.internal_c25f52658a60d1f1fe04d4fdc45521eb/haproxy/0.log" Jul 20 18:40:20.129386 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:20.129346 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b67867474-g8b4g_564a169a-4c89-4367-82f1-788b5cd2fa75/authorino/0.log" Jul 20 18:40:20.162089 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:20.162063 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-z2ptg_b40d9a41-4fcc-48bb-ae51-6acad5a9034d/manager/0.log" Jul 20 18:40:20.199883 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:20.199853 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vq562_f6a9c81e-d746-4fd3-9f19-4dbe5ae8c680/manager/0.log" Jul 20 18:40:20.293546 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:20.293517 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4vs7j_79e7c853-db8a-4810-a9a8-87d588931153/registry-server/0.log" Jul 20 18:40:20.902525 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:20.902489 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pn7sf_dab3ab50-e988-4b19-90fb-2506ed0768f8/manager/0.log" Jul 20 18:40:23.205813 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:23.205780 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8xwb_af3affcc-e4a1-4fb3-9e97-8bb91700d6f6/node-exporter/0.log" Jul 20 18:40:23.231829 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:23.231800 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8xwb_af3affcc-e4a1-4fb3-9e97-8bb91700d6f6/kube-rbac-proxy/0.log" Jul 20 18:40:23.263536 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:23.263509 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n8xwb_af3affcc-e4a1-4fb3-9e97-8bb91700d6f6/init-textfile/0.log" Jul 20 18:40:25.754064 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754032 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh"] Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754389 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754400 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754408 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" containerName="main" Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754414 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" containerName="main" Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754483 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf297bf-2a97-4478-8d6b-c6e22076a934" containerName="istio-proxy" Jul 20 18:40:25.754526 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.754501 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea42ec01-7533-40df-bdba-7a9a994bc9b8" containerName="main" Jul 20 18:40:25.757601 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.757583 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.760523 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.760502 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gtvv\"/\"kube-root-ca.crt\"" Jul 20 18:40:25.760686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.760577 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gtvv\"/\"openshift-service-ca.crt\"" Jul 20 18:40:25.760686 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.760675 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9gtvv\"/\"default-dockercfg-knh4w\"" Jul 20 18:40:25.772849 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.772825 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh"] Jul 20 18:40:25.884567 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.884527 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-podres\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.884772 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.884579 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-lib-modules\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.884772 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.884605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-sys\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.884772 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.884695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-proc\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.884772 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.884712 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7fs\" (UniqueName: \"kubernetes.io/projected/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-kube-api-access-xd7fs\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.985994 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.985957 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-lib-modules\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.985998 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-sys\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986055 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-proc\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7fs\" (UniqueName: \"kubernetes.io/projected/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-kube-api-access-xd7fs\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986157 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-lib-modules\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986194 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986162 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-sys\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986404 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986164 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-podres\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986404 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986161 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-proc\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.986404 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.986237 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-podres\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:25.996031 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:25.996003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7fs\" (UniqueName: \"kubernetes.io/projected/ad7f8b12-0e6b-418e-8712-4d057cf2b4a5-kube-api-access-xd7fs\") pod \"perf-node-gather-daemonset-nd7rh\" (UID: \"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5\") " pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:26.067875 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:26.067783 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:26.209385 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:26.209306 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh"] Jul 20 18:40:26.211976 ip-10-0-135-108 kubenswrapper[2554]: W0720 18:40:26.211940 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podad7f8b12_0e6b_418e_8712_4d057cf2b4a5.slice/crio-df8b3dddbc4ab37e940aed7c04af5fb4f53542f958f09c42195c783659cc3e89 WatchSource:0}: Error finding container df8b3dddbc4ab37e940aed7c04af5fb4f53542f958f09c42195c783659cc3e89: Status 404 returned error can't find the container with id df8b3dddbc4ab37e940aed7c04af5fb4f53542f958f09c42195c783659cc3e89 Jul 20 18:40:27.137766 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:27.137731 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" event={"ID":"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5","Type":"ContainerStarted","Data":"8f9ad06050811d7094d0c77e2ba2b2773469a927b934c901ea4668a6ecc6a634"} Jul 20 18:40:27.138191 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:27.137781 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" event={"ID":"ad7f8b12-0e6b-418e-8712-4d057cf2b4a5","Type":"ContainerStarted","Data":"df8b3dddbc4ab37e940aed7c04af5fb4f53542f958f09c42195c783659cc3e89"} Jul 20 18:40:27.138191 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:27.137867 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:27.157977 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:27.157924 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" podStartSLOduration=2.157909881 podStartE2EDuration="2.157909881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 18:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:40:27.156019161 +0000 UTC m=+3155.094297472" watchObservedRunningTime="2026-07-20 18:40:27.157909881 +0000 UTC m=+3155.096188170" Jul 20 18:40:28.152576 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:28.152538 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t54gs_ff1e5356-333b-49bd-a169-6b2c5f603205/dns/0.log" Jul 20 18:40:28.177591 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:28.177551 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t54gs_ff1e5356-333b-49bd-a169-6b2c5f603205/kube-rbac-proxy/0.log" Jul 20 18:40:28.210684 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:28.210651 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5l5wk_d7dff9a9-ab37-4319-bebc-799e402d6e2c/dns-node-resolver/0.log" Jul 20 18:40:28.839997 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:28.839966 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-78xsn_baa865ba-f651-4070-8e45-026a0bda5a09/node-ca/0.log" Jul 20 18:40:29.871549 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:29.871486 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45wngf_e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9/istio-proxy/0.log" Jul 20 18:40:30.059161 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.059126 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b99948476-qr9bh_8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c/kube-auth-proxy/0.log" Jul 20 18:40:30.089434 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.089395 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/0.log" Jul 20 18:40:30.116954 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.116921 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/1.log" Jul 20 18:40:30.149993 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.149904 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-w9mr7_3f02a0cf-de11-4252-bc3f-30720bb552aa/payload-pre-processing/0.log" Jul 20 18:40:30.181264 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.181235 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-nv7tm_cb9af6ee-cf84-4ab1-91d8-89547006f1ba/payload-processing/0.log" Jul 20 18:40:30.989948 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:30.989921 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nn9qb_fffd3795-b55e-4aa6-a848-934f0f73b203/serve-healthcheck-canary/0.log" Jul 20 18:40:31.674372 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:31.674342 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxtv7_61b66212-57ef-433f-acd2-d05cd43bd46c/kube-rbac-proxy/0.log" Jul 20 18:40:31.700379 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:31.700350 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxtv7_61b66212-57ef-433f-acd2-d05cd43bd46c/exporter/0.log" Jul 20 18:40:31.726533 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:31.726505 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gxtv7_61b66212-57ef-433f-acd2-d05cd43bd46c/extractor/0.log" Jul 20 18:40:33.152466 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:33.152439 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9gtvv/perf-node-gather-daemonset-nd7rh" Jul 20 18:40:33.885897 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:33.885868 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cthmn_818c0036-9703-485a-babb-631ceeb82bfb/manager/0.log" Jul 20 18:40:34.068511 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.068480 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lh76v_dd06f2ad-09ea-4eb4-adaf-cee1749697ec/manager/0.log" Jul 20 18:40:34.181257 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.181174 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-584c885fff-f52zw_0c4e4c8a-8179-4c4f-b4d7-b1a490b5d089/manager/0.log" Jul 20 18:40:34.204292 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.204266 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qp89n_df491736-5bfe-4358-9538-7a3adb0fd84b/server/0.log" Jul 20 18:40:34.275690 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.275657 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-75cmz_d032002b-b768-47e1-b32b-4c42377eede2/manager/0.log" Jul 20 18:40:34.402352 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.402322 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-mnlgv_f753c167-f037-420a-a79b-1843f6f5adf2/manager/0.log" Jul 20 18:40:34.451411 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.451334 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f8f8bd966-mnlgv_f753c167-f037-420a-a79b-1843f6f5adf2/copy-manifests/0.log" Jul 20 18:40:34.748500 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.748408 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45wngf_e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9/istio-proxy/0.log" Jul 20 18:40:34.895349 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.895320 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b99948476-qr9bh_8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c/kube-auth-proxy/0.log" Jul 20 18:40:34.917216 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.917186 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/0.log" Jul 20 18:40:34.941255 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.941226 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/1.log" Jul 20 18:40:34.962464 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.962435 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-w9mr7_3f02a0cf-de11-4252-bc3f-30720bb552aa/payload-pre-processing/0.log" Jul 20 18:40:34.983303 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:34.983278 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-nv7tm_cb9af6ee-cf84-4ab1-91d8-89547006f1ba/payload-processing/0.log" Jul 20 18:40:35.973150 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:35.973120 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zvw7l_db393466-012a-461f-be36-f39b92fcd94d/openshift-lws-operator/0.log" Jul 20 18:40:36.346059 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.346036 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45wngf_e38e833b-a5c0-4bd7-b2d9-9dc7d85107a9/istio-proxy/0.log" Jul 20 18:40:36.508143 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.508114 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b99948476-qr9bh_8f5f73c6-c4a3-4a59-9e68-0e1d64b18f3c/kube-auth-proxy/0.log" Jul 20 18:40:36.539977 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.539949 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/0.log" Jul 20 18:40:36.565605 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.565568 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-tlcgp_ee786b93-3864-4931-8670-b7b1736db018/istio-proxy/1.log" Jul 20 18:40:36.590393 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.590323 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-w9mr7_3f02a0cf-de11-4252-bc3f-30720bb552aa/payload-pre-processing/0.log" Jul 20 18:40:36.611773 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:36.611745 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-nv7tm_cb9af6ee-cf84-4ab1-91d8-89547006f1ba/payload-processing/0.log" Jul 20 18:40:42.970270 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:42.970243 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/kube-multus-additional-cni-plugins/0.log" Jul 20 18:40:43.007716 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.007681 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/egress-router-binary-copy/0.log" Jul 20 18:40:43.042112 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.042086 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/cni-plugins/0.log" Jul 20 18:40:43.081933 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.081909 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/bond-cni-plugin/0.log" Jul 20 18:40:43.122110 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.122081 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/routeoverride-cni/0.log" Jul 20 18:40:43.157864 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.157840 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/whereabouts-cni-bincopy/0.log" Jul 20 18:40:43.196361 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.196339 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r2vz_fa6ae863-c428-499f-8d98-a2a8b36edfeb/whereabouts-cni/0.log" Jul 20 18:40:43.290643 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.290593 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mrspd_498e6195-611a-45bf-a575-8d80a58307b8/kube-multus/0.log" Jul 20 18:40:43.399898 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.399869 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-46fm2_084c96d0-3337-4ff6-8122-73354392e16c/network-metrics-daemon/0.log" Jul 20 18:40:43.441150 ip-10-0-135-108 kubenswrapper[2554]: I0720 18:40:43.441127 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-46fm2_084c96d0-3337-4ff6-8122-73354392e16c/kube-rbac-proxy/0.log"