Jul 24 22:12:47.857666 ip-10-0-132-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 22:12:47.857679 ip-10-0-132-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 22:12:47.857689 ip-10-0-132-132 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 22:12:47.858007 ip-10-0-132-132 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 22:12:58.106541 ip-10-0-132-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 22:12:58.106557 ip-10-0-132-132 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e60d1cb4e2b542aeb7cc1c3dedc331b8 -- Jul 24 22:14:37.199305 ip-10-0-132-132 systemd[1]: Starting Kubernetes Kubelet... Jul 24 22:14:37.676235 ip-10-0-132-132 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:37.676235 ip-10-0-132-132 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 22:14:37.676235 ip-10-0-132-132 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:37.676235 ip-10-0-132-132 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 22:14:37.676235 ip-10-0-132-132 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:14:37.676866 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.676282 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 22:14:37.682187 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682167 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.682187 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682184 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.682187 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682188 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.682187 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682191 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.682187 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682194 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682197 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682200 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682203 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682206 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682208 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682211 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682214 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682217 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682219 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682222 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682224 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682227 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682230 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682232 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682235 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682238 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682240 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682242 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.682373 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682245 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682247 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682250 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682252 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682254 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682260 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682262 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682265 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682267 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682269 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682272 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682274 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682277 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682279 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682281 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682285 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682288 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682292 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682295 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.682856 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682298 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682301 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682304 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682307 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682309 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682312 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682316 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682319 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682321 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682324 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682327 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682329 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682331 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682334 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682336 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682340 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682343 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682346 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682350 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682352 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.683334 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682355 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682357 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682360 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682362 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682365 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682367 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682369 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682372 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682375 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682377 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682380 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682383 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682386 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682388 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682391 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682393 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682396 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682398 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682402 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682404 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.683923 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682407 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682409 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682411 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682414 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682416 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682418 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682421 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682423 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682427 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682430 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682433 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682435 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682437 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682440 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682442 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682444 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.682447 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683148 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683155 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.684415 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683157 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683160 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683163 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683166 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683169 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683171 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683174 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683176 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683179 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683181 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683183 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683186 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683188 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683191 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683194 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683196 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683199 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683201 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683204 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683206 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.684865 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683209 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683211 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683213 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683216 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683219 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683221 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683224 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683226 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683228 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683231 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683234 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683236 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683239 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683242 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683244 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683247 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683250 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683252 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683255 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.685337 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683257 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683260 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683262 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683264 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683267 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683269 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683272 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683274 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683277 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683279 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683281 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683284 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683286 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683289 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683291 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683313 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683316 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683319 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683322 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683325 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.685824 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683328 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683330 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683333 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683336 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683339 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683341 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683346 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683349 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683351 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683355 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683359 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683362 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683365 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683367 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683371 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683375 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683378 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683381 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683383 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.686285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683387 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683389 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683392 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683394 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683397 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683399 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683402 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683404 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683407 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683410 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683414 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683418 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683421 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683424 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683428 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683430 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683433 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683435 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683437 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683825 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.686743 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683830 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683833 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683835 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683838 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683840 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683843 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683845 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683848 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683850 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683853 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683855 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683858 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683861 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683864 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683866 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683869 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683872 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683876 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683878 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683881 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.687263 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683884 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683887 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683891 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683894 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683897 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683900 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683902 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683905 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683908 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683910 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683913 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683915 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683918 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683920 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683923 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683925 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683928 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683931 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683934 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683936 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.687763 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683939 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683942 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683944 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683947 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683949 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683952 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683955 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683957 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683959 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683962 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683964 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683967 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683970 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683972 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683975 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683990 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683993 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683995 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.683997 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.688236 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684000 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684002 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684004 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684007 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684009 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684013 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684017 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684020 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684022 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684025 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684028 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684031 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684034 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684036 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684039 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684041 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684044 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684046 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684048 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.688679 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684052 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684054 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684056 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684059 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684062 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684064 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684069 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684071 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684074 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684076 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684079 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684082 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684084 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684087 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684089 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684093 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684095 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684097 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684100 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684102 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684166 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 22:14:37.689108 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684173 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684178 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684182 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684186 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684189 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684193 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684197 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684199 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684202 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684205 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684208 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684211 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684215 2576 flags.go:64] FLAG: --cgroup-root="" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684218 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684220 2576 flags.go:64] FLAG: --client-ca-file="" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684223 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684226 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684231 2576 flags.go:64] FLAG: --cluster-domain="" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684233 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684236 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684239 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684242 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684245 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 22:14:37.689641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684248 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684251 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684254 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684258 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684261 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684264 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684268 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684270 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684274 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684276 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684279 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684282 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684284 2576 flags.go:64] FLAG: --enable-server="true" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684287 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684292 2576 flags.go:64] FLAG: --event-burst="100" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684294 2576 flags.go:64] FLAG: --event-qps="50" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684297 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684300 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684303 2576 flags.go:64] FLAG: --eviction-hard="" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684306 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684309 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684311 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684315 2576 flags.go:64] FLAG: --eviction-soft="" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684317 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684320 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 22:14:37.690164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684323 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684327 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684330 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684332 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684335 2576 flags.go:64] FLAG: --feature-gates="" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684339 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684342 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684344 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684347 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684350 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684355 2576 flags.go:64] FLAG: --help="false" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684357 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-132-132.ec2.internal" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684360 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684363 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684366 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684369 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684372 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684375 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684378 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684380 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684383 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684385 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684388 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684391 2576 flags.go:64] FLAG: --kube-reserved="" Jul 24 22:14:37.690805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684394 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684396 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684399 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684401 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684404 2576 flags.go:64] FLAG: --lock-file="" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684407 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684410 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684413 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684418 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684423 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684426 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684429 2576 flags.go:64] FLAG: --logging-format="text" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684431 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684435 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684437 2576 flags.go:64] FLAG: --manifest-url="" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684440 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684444 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684446 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684451 2576 flags.go:64] FLAG: --max-pods="110" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684454 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684457 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684460 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684463 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684465 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684468 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 22:14:37.691434 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684471 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684477 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684479 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684482 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684485 2576 flags.go:64] FLAG: --pod-cidr="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684488 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684492 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684495 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684498 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684500 2576 flags.go:64] FLAG: --port="10250" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684504 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684506 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0043e059fac67b362" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684509 2576 flags.go:64] FLAG: --qos-reserved="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684512 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684515 2576 flags.go:64] FLAG: --register-node="true" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684518 2576 flags.go:64] FLAG: --register-with-taints="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684521 2576 flags.go:64] FLAG: --registry-burst="10" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684524 2576 flags.go:64] FLAG: --registry-qps="5" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684527 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684530 2576 flags.go:64] FLAG: --reserved-memory="" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684533 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684536 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684539 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684541 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684544 2576 flags.go:64] FLAG: --runonce="false" Jul 24 22:14:37.692060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684547 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684564 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684567 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684569 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684572 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684575 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684577 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684580 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684583 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684585 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684588 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684591 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684593 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684596 2576 flags.go:64] FLAG: --system-cgroups="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684599 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684604 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684606 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684609 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684613 2576 flags.go:64] FLAG: --tls-min-version="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684615 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684617 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684620 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684623 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684626 2576 flags.go:64] FLAG: --v="2" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684630 2576 flags.go:64] FLAG: --version="false" Jul 24 22:14:37.692640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684634 2576 flags.go:64] FLAG: --vmodule="" Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684637 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.684640 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684760 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684764 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684768 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684771 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684775 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684778 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684781 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684783 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684785 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684788 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684790 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684793 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684797 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684800 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684803 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684806 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684809 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.693224 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684812 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684814 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684817 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684819 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684822 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684825 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684827 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684830 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684832 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684835 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684839 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684842 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684845 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684847 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684850 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684852 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684855 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684857 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684859 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684863 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.693713 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684865 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684868 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684870 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684873 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684875 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684878 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684880 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684883 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684885 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684888 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684890 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684892 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684896 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684899 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684902 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684905 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684907 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684910 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684913 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.694172 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684915 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684917 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684920 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684922 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684925 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684927 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684930 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684932 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684934 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684937 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684939 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684941 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684948 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684950 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684953 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684955 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684957 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684960 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684962 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.694654 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684964 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684967 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684969 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684972 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684974 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684977 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684979 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684981 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684984 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684986 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684989 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684991 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684993 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684996 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.684998 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685000 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685003 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685007 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685010 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685012 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.695136 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685015 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685017 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685020 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.685022 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.686126 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.692626 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.692638 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.692657 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.692662 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692717 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692721 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692724 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692727 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692730 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692732 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.695674 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692735 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692737 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692740 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692742 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692745 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692747 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692750 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692753 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692755 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692758 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692760 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692762 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692765 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692767 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692769 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692772 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692775 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692777 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692779 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692782 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.696068 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692784 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692786 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692789 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692792 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692795 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692797 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692800 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692802 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692805 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692807 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692809 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692811 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692816 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692819 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692822 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692825 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692828 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692831 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692834 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.696528 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692836 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692840 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692843 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692846 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692848 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692851 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692853 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692856 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692859 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692861 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692863 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692866 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692868 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692870 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692873 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692875 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692878 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692881 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692883 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.697002 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692886 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692889 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692891 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692894 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692896 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692898 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692901 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692903 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692905 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692908 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692910 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692913 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692915 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692917 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692919 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692922 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692924 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692926 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692928 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.697457 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692930 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692933 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692935 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692938 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692940 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692942 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692944 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692947 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692949 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692951 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692953 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692956 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692960 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692963 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692966 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.697907 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.692969 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.692973 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693076 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693080 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693083 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693085 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693088 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693091 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693093 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693096 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693098 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693100 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693103 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693106 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693108 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:14:37.698255 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693110 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693113 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693115 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693117 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693120 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693122 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693124 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693127 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693129 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693131 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693134 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693137 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693139 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693141 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693144 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693147 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693149 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693151 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693154 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693156 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:14:37.698691 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693159 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693161 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693163 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693166 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693168 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693170 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693173 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693175 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693178 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693180 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693183 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693185 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693188 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693190 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693193 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693195 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693197 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693199 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693202 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693204 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:14:37.699202 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693207 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693210 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693213 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693216 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693218 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693220 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693223 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693225 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693228 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693231 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693233 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693236 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693238 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693240 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693243 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693246 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693248 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693250 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693255 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693257 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:14:37.699700 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693260 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693262 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693264 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693267 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693269 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693272 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693274 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693277 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693281 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693284 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693286 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693289 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693292 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693294 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693296 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693299 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693301 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693304 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693306 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:14:37.700175 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693308 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693311 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693315 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693317 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693320 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693322 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:37.693325 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.693329 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.693430 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.695405 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.696802 2576 server.go:1019] "Starting client certificate rotation" Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.696900 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.697781 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:14:37.700731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.699289 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:14:37.702218 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.702201 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:14:37.711883 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.711867 2576 log.go:25] "Validated CRI v1 runtime API" Jul 24 22:14:37.713969 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.713953 2576 log.go:25] "Validated CRI v1 image API" Jul 24 22:14:37.716975 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.716959 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 22:14:37.720511 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.720492 2576 fs.go:135] Filesystem UUIDs: map[52d40337-1061-4349-8be3-5750f87f34e1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bdfb3dad-a44b-46a2-84ec-f1606130b713:/dev/nvme0n1p3] Jul 24 22:14:37.720579 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.720510 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 22:14:37.725428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.725301 2576 manager.go:216] Machine: {Timestamp:2026-07-24 22:14:37.724017948 +0000 UTC m=+0.391028761 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200261 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255c5cb4746c92dc84f8127a4c9ba3 SystemUUID:ec255c5c-b474-6c92-dc84-f8127a4c9ba3 BootID:e60d1cb4-e2b5-42ae-b7cc-1c3dedc331b8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:cf:92:ba:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:cf:92:ba:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:53:d6:69:4f:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 22:14:37.725428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.725424 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 22:14:37.725538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.725513 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 22:14:37.728520 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.728494 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 22:14:37.728664 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.728523 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 22:14:37.729247 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.729232 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:14:37.729450 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.729440 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 22:14:37.729486 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.729452 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 22:14:37.729486 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.729460 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:14:37.730200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.730190 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:14:37.730965 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.730956 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 22:14:37.732626 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732615 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:14:37.732734 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732725 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 22:14:37.732771 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732760 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 24 22:14:37.732797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732773 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 22:14:37.732797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732787 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 22:14:37.732797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732796 2576 kubelet.go:398] "Adding apiserver pod source" Jul 24 22:14:37.732888 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.732807 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 22:14:37.733105 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.733094 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:14:37.733144 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.733111 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:14:37.735163 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.735143 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 22:14:37.736976 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.736963 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 22:14:37.737021 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.736993 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 22:14:37.738417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738405 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738421 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738427 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738433 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738442 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738447 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738454 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 22:14:37.738459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738459 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 22:14:37.738691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738465 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 22:14:37.738691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738470 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 22:14:37.738691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738479 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 22:14:37.738691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.738484 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 22:14:37.739368 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.739359 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 22:14:37.739368 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.739368 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 22:14:37.742781 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.742762 2576 server.go:1268] "Started kubelet" Jul 24 22:14:37.743363 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.743206 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 22:14:37.743363 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.743219 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 22:14:37.743363 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.743286 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 22:14:37.743526 ip-10-0-132-132 systemd[1]: Started Kubernetes Kubelet. Jul 24 22:14:37.743845 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.743765 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 22:14:37.744594 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.744520 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 22:14:37.744594 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.744525 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 22:14:37.745430 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.745411 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 22:14:37.745753 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.745727 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 24 22:14:37.750157 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.750123 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 22:14:37.753488 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.752248 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-132.ec2.internal.18c5598a4655902a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-132.ec2.internal,UID:ip-10-0-132-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-132.ec2.internal,},FirstTimestamp:2026-07-24 22:14:37.742641194 +0000 UTC m=+0.409652005,LastTimestamp:2026-07-24 22:14:37.742641194 +0000 UTC m=+0.409652005,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-132.ec2.internal,}" Jul 24 22:14:37.759500 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.759338 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 22:14:37.759500 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.759351 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 22:14:37.759500 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.759422 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:37.759965 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.759935 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 22:14:37.760074 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760046 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 22:14:37.760074 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760064 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 22:14:37.760231 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760213 2576 factory.go:55] Registering systemd factory Jul 24 22:14:37.760321 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760240 2576 factory.go:223] Registration of the systemd container factory successfully Jul 24 22:14:37.760368 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760351 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 24 22:14:37.760368 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760358 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 24 22:14:37.760527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760450 2576 factory.go:153] Registering CRI-O factory Jul 24 22:14:37.760527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760466 2576 factory.go:223] Registration of the crio container factory successfully Jul 24 22:14:37.760527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760510 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 22:14:37.760688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760534 2576 factory.go:103] Registering Raw factory Jul 24 22:14:37.760688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760579 2576 manager.go:1196] Started watching for new ooms in manager Jul 24 22:14:37.760998 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.760983 2576 manager.go:349] Starting recovery of all containers Jul 24 22:14:37.771126 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.771112 2576 manager.go:354] Recovery completed Jul 24 22:14:37.771655 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.771624 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 22:14:37.771734 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.771651 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 22:14:37.775294 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.775279 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.777006 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.776978 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zpfhs" Jul 24 22:14:37.777082 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777057 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.777136 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777121 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.777174 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777141 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.777670 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777652 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 22:14:37.777670 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777668 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 22:14:37.777795 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777679 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.777795 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.777699 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:14:37.778527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.778512 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.778652 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.778541 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.778652 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.778581 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.779322 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.779309 2576 policy_none.go:49] "None policy: Start" Jul 24 22:14:37.779420 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.779325 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 22:14:37.779420 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.779335 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 22:14:37.780468 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.780457 2576 policy_none.go:47] "Start" Jul 24 22:14:37.783284 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.783250 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zpfhs" Jul 24 22:14:37.787865 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.787803 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-132.ec2.internal.18c5598a4862e382 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-132.ec2.internal,UID:ip-10-0-132-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-132.ec2.internal,},FirstTimestamp:2026-07-24 22:14:37.77706893 +0000 UTC m=+0.444079744,LastTimestamp:2026-07-24 22:14:37.77706893 +0000 UTC m=+0.444079744,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-132.ec2.internal,}" Jul 24 22:14:37.817127 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817112 2576 manager.go:337] "Starting Device Plugin manager" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.817149 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817162 2576 server.go:87] "Starting device plugin registration server" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817421 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817433 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817530 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817628 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.817639 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.818105 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 22:14:37.826092 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.818142 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:37.882408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.882376 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 22:14:37.883770 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.883746 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 22:14:37.883880 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.883793 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 22:14:37.883880 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.883819 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 22:14:37.883880 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.883860 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 22:14:37.886346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.886329 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:37.918484 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.918463 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.919448 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.919434 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.919523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.919464 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.919523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.919473 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.919523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.919496 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-132.ec2.internal" Jul 24 22:14:37.931345 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.931289 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-132.ec2.internal" Jul 24 22:14:37.931345 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.931314 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-132.ec2.internal\": node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:37.959155 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:37.959132 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:37.984294 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.984270 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal"] Jul 24 22:14:37.984383 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.984341 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.985236 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.985223 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.985287 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.985254 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.985287 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.985266 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.986537 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.986523 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.986676 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.986656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:37.986716 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.986697 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.987272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987250 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.987272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987273 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.987417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987287 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.987417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987255 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.987417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987354 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.987417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.987370 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:37.988371 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.988356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:37.988449 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.988380 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:37.989163 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.989136 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:37.989255 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.989169 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:37.989255 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:37.989179 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:38.010363 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.010333 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-132.ec2.internal\" not found" node="ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.014747 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.014730 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-132.ec2.internal\" not found" node="ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.059725 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.059693 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.061912 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.061896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.061975 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.061922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.061975 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.061941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb745cffc865678816fc0718fd4a2e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-132.ec2.internal\" (UID: \"1eb745cffc865678816fc0718fd4a2e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.160371 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.160348 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.162528 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.162619 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.162619 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb745cffc865678816fc0718fd4a2e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-132.ec2.internal\" (UID: \"1eb745cffc865678816fc0718fd4a2e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.162704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162614 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.162704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb745cffc865678816fc0718fd4a2e2-config\") pod \"kube-apiserver-proxy-ip-10-0-132-132.ec2.internal\" (UID: \"1eb745cffc865678816fc0718fd4a2e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.162704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.162679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8256975cb148fd9685d00b166788932a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal\" (UID: \"8256975cb148fd9685d00b166788932a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.261135 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.261062 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.313403 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.313367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.318120 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.318097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:38.361614 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.361417 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.461984 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.461947 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.562541 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.562462 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.663117 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.663079 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.697424 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.697382 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 22:14:38.697894 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.697574 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:14:38.763727 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.763695 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.788373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.788332 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 22:09:38 +0000 UTC" deadline="2028-02-01 03:28:43.138388412 +0000 UTC" Jul 24 22:14:38.788373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.788365 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13349h14m4.35002644s" Jul 24 22:14:38.863967 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.863916 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:38.929713 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.929681 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:14:38.931329 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.931304 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:14:38.931450 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.931340 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:14:38.931450 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.931354 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:14:38.945443 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:38.945404 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb745cffc865678816fc0718fd4a2e2.slice/crio-c88aeb4f75d78088d81ee483568441c500742d0479582bb48d89abfc03ce03c4 WatchSource:0}: Error finding container c88aeb4f75d78088d81ee483568441c500742d0479582bb48d89abfc03ce03c4: Status 404 returned error can't find the container with id c88aeb4f75d78088d81ee483568441c500742d0479582bb48d89abfc03ce03c4 Jul 24 22:14:38.945772 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:38.945751 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8256975cb148fd9685d00b166788932a.slice/crio-6869f5eca18342cc66c9ff57f8344f2765bfd036827e45d71588923833cb71d4 WatchSource:0}: Error finding container 6869f5eca18342cc66c9ff57f8344f2765bfd036827e45d71588923833cb71d4: Status 404 returned error can't find the container with id 6869f5eca18342cc66c9ff57f8344f2765bfd036827e45d71588923833cb71d4 Jul 24 22:14:38.951217 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:38.951198 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:14:38.964608 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:38.964588 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:39.000085 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.000049 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:39.057914 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.057882 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:39.065445 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.065421 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-132.ec2.internal\" not found" Jul 24 22:14:39.100876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.100857 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:39.160510 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.160460 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" Jul 24 22:14:39.173514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.173487 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:14:39.174563 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.174535 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" Jul 24 22:14:39.182856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.182838 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:14:39.533847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.533766 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:14:39.734410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.734375 2576 apiserver.go:52] "Watching apiserver" Jul 24 22:14:39.740911 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.740885 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 22:14:39.741473 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.741448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6zjk2","openshift-image-registry/node-ca-bjl4j","openshift-multus/multus-8f4qr","openshift-multus/multus-additional-cni-plugins-b8kmm","openshift-multus/network-metrics-daemon-7vhmh","kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal","openshift-cluster-node-tuning-operator/tuned-l2gz2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal","openshift-network-diagnostics/network-check-target-vcvw9","openshift-network-operator/iptables-alerter-x6rf9","openshift-ovn-kubernetes/ovnkube-node-4pkmq","kube-system/global-pull-secret-syncer-vpxvh","kube-system/konnectivity-agent-zzhmd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf"] Jul 24 22:14:39.743291 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.743268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.744388 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.744366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.745503 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.745482 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.745611 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.745526 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.745611 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.745548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z7f8v\"" Jul 24 22:14:39.746117 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.746095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.747028 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.746619 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 22:14:39.747028 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.746646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 22:14:39.747028 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.746889 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2jwq2\"" Jul 24 22:14:39.747398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.747379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.748837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.748093 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 22:14:39.748837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.748235 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 22:14:39.748837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.748324 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.748837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.748680 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.748837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.748805 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-87bsr\"" Jul 24 22:14:39.749207 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.749183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.749282 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.749244 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 22:14:39.749282 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.749257 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:39.749391 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.749298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qfdvw\"" Jul 24 22:14:39.749868 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.749693 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 22:14:39.750830 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.750810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.753688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.753183 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 22:14:39.753688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.753319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v9qb\"" Jul 24 22:14:39.753688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.753467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.753688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.753538 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.753688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.753669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:39.754009 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.753739 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:39.754869 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.754743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.757690 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.757409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgvmh\"" Jul 24 22:14:39.758624 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.758592 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.760238 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.759460 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 22:14:39.760238 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.759485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.761142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.760506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.761502 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.761430 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.762385 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.762349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.762768 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.762499 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 22:14:39.762768 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.762606 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgp47\"" Jul 24 22:14:39.763504 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.763456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.765868 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.765397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.766134 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.766115 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 22:14:39.766215 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.766142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dvp2c\"" Jul 24 22:14:39.766353 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.766333 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.766423 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.766369 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.766949 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.766932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.767048 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.767002 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:39.767428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.767408 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 22:14:39.767756 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.767705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5c2mb\"" Jul 24 22:14:39.767756 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.767727 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 22:14:39.770854 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.770835 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 22:14:39.770954 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.770838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.772639 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.772913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-multus-certs\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.772950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.772955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-os-release\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.772988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0824b1c-cbf0-4438-afa9-b628cb1b4203-hosts-file\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849md\" (UniqueName: \"kubernetes.io/projected/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-kube-api-access-849md\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35046547-570e-4b84-bd6d-12d666d33d7d-host-slash\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-hostroot\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-systemd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-agent-certs\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-os-release\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5c46\" (UniqueName: \"kubernetes.io/projected/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-kube-api-access-m5c46\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysconfig\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-conf\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-tuned\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35046547-570e-4b84-bd6d-12d666d33d7d-iptables-alerter-script\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjgj\" (UniqueName: \"kubernetes.io/projected/35046547-570e-4b84-bd6d-12d666d33d7d-kube-api-access-pmjgj\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-konnectivity-ca\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-socket-dir-parent\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-multus\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-kubelet\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-device-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-modprobe-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-run\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-cnibin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.773847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cnibin\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4czd\" (UniqueName: \"kubernetes.io/projected/c67a50a8-0662-4102-8bcf-1e78a529a291-kube-api-access-w4czd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-cni-binary-copy\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frn7t\" (UniqueName: \"kubernetes.io/projected/780fb566-4aca-4c9c-8c04-e794f1a4472e-kube-api-access-frn7t\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0824b1c-cbf0-4438-afa9-b628cb1b4203-tmp-dir\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd98l\" (UniqueName: \"kubernetes.io/projected/f0824b1c-cbf0-4438-afa9-b628cb1b4203-kube-api-access-pd98l\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-var-lib-kubelet\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-etc-selinux\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-system-cni-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-socket-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.773997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-kubernetes\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-host\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-tmp\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-sys-fs\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-bin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.774538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-lib-modules\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-system-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-registration-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-host\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-conf-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-etc-kubernetes\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6ct\" (UniqueName: \"kubernetes.io/projected/56481edd-b00d-48c6-88c7-b41cc816ad04-kube-api-access-rc6ct\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-serviceca\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbrv\" (UniqueName: \"kubernetes.io/projected/f1654044-8118-4cc6-8743-397a4adab49f-kube-api-access-spbrv\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-k8s-cni-cncf-io\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-daemon-config\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.775467 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.776392 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-sys\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.776392 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.774686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-netns\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.777648 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.777624 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:14:39.807118 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.807092 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2g6fm" Jul 24 22:14:39.817875 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.816837 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2g6fm" Jul 24 22:14:39.865708 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.861659 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 22:14:39.875484 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-systemd-units\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-env-overrides\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-serviceca\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.875750 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbrv\" (UniqueName: \"kubernetes.io/projected/f1654044-8118-4cc6-8743-397a4adab49f-kube-api-access-spbrv\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.875867 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:14:40.375823067 +0000 UTC m=+3.042833891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-netd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.875979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.875962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-k8s-cni-cncf-io\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-daemon-config\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-sys\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-slash\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-netns\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-multus-certs\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-kubelet-config\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-ovn\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46cdfd87-1536-465a-9409-08538c82df55-ovn-node-metrics-cert\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-log-socket\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-os-release\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0824b1c-cbf0-4438-afa9-b628cb1b4203-hosts-file\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849md\" (UniqueName: \"kubernetes.io/projected/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-kube-api-access-849md\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35046547-570e-4b84-bd6d-12d666d33d7d-host-slash\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-var-lib-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-config\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-hostroot\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-systemd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-agent-certs\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-daemon-config\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-serviceca\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cni-binary-copy\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.876991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-os-release\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-multus-certs\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-netns\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-run-k8s-cni-cncf-io\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-os-release\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/35046547-570e-4b84-bd6d-12d666d33d7d-host-slash\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c46\" (UniqueName: \"kubernetes.io/projected/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-kube-api-access-m5c46\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-systemd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.876998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-hostroot\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysconfig\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-conf\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0824b1c-cbf0-4438-afa9-b628cb1b4203-hosts-file\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-tuned\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877056 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35046547-570e-4b84-bd6d-12d666d33d7d-iptables-alerter-script\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjgj\" (UniqueName: \"kubernetes.io/projected/35046547-570e-4b84-bd6d-12d666d33d7d-kube-api-access-pmjgj\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-konnectivity-ca\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.877876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysconfig\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-socket-dir-parent\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-multus\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-kubelet\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-device-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-os-release\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877273 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-sys\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-sysctl-conf\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-socket-dir-parent\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-modprobe-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-run\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-cnibin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cnibin\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-multus\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4czd\" (UniqueName: \"kubernetes.io/projected/c67a50a8-0662-4102-8bcf-1e78a529a291-kube-api-access-w4czd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-run\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.878813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-device-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-dbus\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-kubelet\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlq4h\" (UniqueName: \"kubernetes.io/projected/46cdfd87-1536-465a-9409-08538c82df55-kube-api-access-vlq4h\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-cni-binary-copy\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frn7t\" (UniqueName: \"kubernetes.io/projected/780fb566-4aca-4c9c-8c04-e794f1a4472e-kube-api-access-frn7t\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0824b1c-cbf0-4438-afa9-b628cb1b4203-tmp-dir\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd98l\" (UniqueName: \"kubernetes.io/projected/f0824b1c-cbf0-4438-afa9-b628cb1b4203-kube-api-access-pd98l\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-konnectivity-ca\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-var-lib-kubelet\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-netns\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-etc-selinux\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-node-log\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-system-cni-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-socket-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-kubernetes\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.879602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-host\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-tmp\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-etc-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-sys-fs\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-kubelet\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-systemd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.877583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-modprobe-d\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-var-lib-kubelet\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/35046547-570e-4b84-bd6d-12d666d33d7d-iptables-alerter-script\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-etc-selinux\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-system-cni-dir\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-socket-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-sys-fs\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-kubernetes\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-host\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56481edd-b00d-48c6-88c7-b41cc816ad04-cni-binary-copy\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-cnibin\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.880230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-cnibin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-script-lib\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-bin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878754 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-host-var-lib-cni-bin\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-lib-modules\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-system-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.878963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-registration-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c67a50a8-0662-4102-8bcf-1e78a529a291-lib-modules\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/780fb566-4aca-4c9c-8c04-e794f1a4472e-registration-dir\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-host\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-system-cni-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-bin\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-host\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-conf-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.880865 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-etc-kubernetes\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.881428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6ct\" (UniqueName: \"kubernetes.io/projected/56481edd-b00d-48c6-88c7-b41cc816ad04-kube-api-access-rc6ct\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.881428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-multus-conf-dir\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.881428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56481edd-b00d-48c6-88c7-b41cc816ad04-etc-kubernetes\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.881428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.879241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0824b1c-cbf0-4438-afa9-b628cb1b4203-tmp-dir\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.882506 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.882327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-tmp\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.882506 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.882421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c67a50a8-0662-4102-8bcf-1e78a529a291-etc-tuned\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.882722 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.882572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9943b7f9-8c27-4004-ab2b-f21eeef6bf73-agent-certs\") pod \"konnectivity-agent-zzhmd\" (UID: \"9943b7f9-8c27-4004-ab2b-f21eeef6bf73\") " pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:39.889699 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.889305 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:39.889699 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.889327 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:39.889699 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.889341 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:39.889699 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.889405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:14:40.389385155 +0000 UTC m=+3.056395979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:39.889699 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.889662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4czd\" (UniqueName: \"kubernetes.io/projected/c67a50a8-0662-4102-8bcf-1e78a529a291-kube-api-access-w4czd\") pod \"tuned-l2gz2\" (UID: \"c67a50a8-0662-4102-8bcf-1e78a529a291\") " pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:39.892657 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.891486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" event={"ID":"1eb745cffc865678816fc0718fd4a2e2","Type":"ContainerStarted","Data":"c88aeb4f75d78088d81ee483568441c500742d0479582bb48d89abfc03ce03c4"} Jul 24 22:14:39.893336 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.892904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-849md\" (UniqueName: \"kubernetes.io/projected/0a3330cf-b271-47f3-8cb8-80eea9cfaa50-kube-api-access-849md\") pod \"node-ca-bjl4j\" (UID: \"0a3330cf-b271-47f3-8cb8-80eea9cfaa50\") " pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:39.893336 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.893259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c46\" (UniqueName: \"kubernetes.io/projected/4689ab96-2a41-4e2e-8f6b-243f97b29cdc-kube-api-access-m5c46\") pod \"multus-additional-cni-plugins-b8kmm\" (UID: \"4689ab96-2a41-4e2e-8f6b-243f97b29cdc\") " pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:39.893841 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.893817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" event={"ID":"8256975cb148fd9685d00b166788932a","Type":"ContainerStarted","Data":"6869f5eca18342cc66c9ff57f8344f2765bfd036827e45d71588923833cb71d4"} Jul 24 22:14:39.894971 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.894952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frn7t\" (UniqueName: \"kubernetes.io/projected/780fb566-4aca-4c9c-8c04-e794f1a4472e-kube-api-access-frn7t\") pod \"aws-ebs-csi-driver-node-g49bf\" (UID: \"780fb566-4aca-4c9c-8c04-e794f1a4472e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:39.895520 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.895492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjgj\" (UniqueName: \"kubernetes.io/projected/35046547-570e-4b84-bd6d-12d666d33d7d-kube-api-access-pmjgj\") pod \"iptables-alerter-x6rf9\" (UID: \"35046547-570e-4b84-bd6d-12d666d33d7d\") " pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:39.896283 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.896252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd98l\" (UniqueName: \"kubernetes.io/projected/f0824b1c-cbf0-4438-afa9-b628cb1b4203-kube-api-access-pd98l\") pod \"node-resolver-6zjk2\" (UID: \"f0824b1c-cbf0-4438-afa9-b628cb1b4203\") " pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:39.896368 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.896345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbrv\" (UniqueName: \"kubernetes.io/projected/f1654044-8118-4cc6-8743-397a4adab49f-kube-api-access-spbrv\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:39.898177 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.898150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6ct\" (UniqueName: \"kubernetes.io/projected/56481edd-b00d-48c6-88c7-b41cc816ad04-kube-api-access-rc6ct\") pod \"multus-8f4qr\" (UID: \"56481edd-b00d-48c6-88c7-b41cc816ad04\") " pod="openshift-multus/multus-8f4qr" Jul 24 22:14:39.979663 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-systemd-units\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-env-overrides\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-systemd-units\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-netd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-netd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.979839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-slash\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-kubelet-config\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-slash\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-ovn\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.979888 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46cdfd87-1536-465a-9409-08538c82df55-ovn-node-metrics-cert\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-kubelet-config\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-ovn\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:39.979951 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:40.479930104 +0000 UTC m=+3.146940928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.979974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-log-socket\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-var-lib-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-config\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980190 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-var-lib-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-log-socket\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-dbus\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.980260 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-kubelet\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlq4h\" (UniqueName: \"kubernetes.io/projected/46cdfd87-1536-465a-9409-08538c82df55-kube-api-access-vlq4h\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-netns\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-env-overrides\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-node-log\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-etc-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-kubelet\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-systemd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-script-lib\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-dbus\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-node-log\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-bin\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-netns\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-run-systemd\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980509 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-etc-openvswitch\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-run-ovn-kubernetes\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46cdfd87-1536-465a-9409-08538c82df55-host-cni-bin\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.980935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-config\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.981742 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.981020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46cdfd87-1536-465a-9409-08538c82df55-ovnkube-script-lib\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.983258 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.983237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46cdfd87-1536-465a-9409-08538c82df55-ovn-node-metrics-cert\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:39.991709 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:39.991686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlq4h\" (UniqueName: \"kubernetes.io/projected/46cdfd87-1536-465a-9409-08538c82df55-kube-api-access-vlq4h\") pod \"ovnkube-node-4pkmq\" (UID: \"46cdfd87-1536-465a-9409-08538c82df55\") " pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:40.056599 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.056490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" Jul 24 22:14:40.065759 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.065729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:14:40.078025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.078003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zjk2" Jul 24 22:14:40.084256 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.084236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8f4qr" Jul 24 22:14:40.094085 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.094055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" Jul 24 22:14:40.102094 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.102068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" Jul 24 22:14:40.109752 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.109706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjl4j" Jul 24 22:14:40.119616 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.119589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6rf9" Jul 24 22:14:40.124728 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.124705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:14:40.383806 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.383731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:40.384009 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.383864 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:40.384009 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.383946 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:14:41.383925539 +0000 UTC m=+4.050936359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:40.485073 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.485025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:40.485274 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.485095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:40.485274 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485208 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:40.485274 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485237 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:40.485274 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485264 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:40.485480 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485279 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:40.485480 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485280 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:41.485262659 +0000 UTC m=+4.152273465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:40.485480 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.485352 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:14:41.485333078 +0000 UTC m=+4.152343888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:40.819843 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.819718 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:09:40 +0000 UTC" deadline="2028-02-05 22:09:09.345690843 +0000 UTC" Jul 24 22:14:40.819843 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.819783 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13463h54m28.525911439s" Jul 24 22:14:40.884293 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:40.884250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:40.884489 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:40.884392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:40.939516 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.938919 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56481edd_b00d_48c6_88c7_b41cc816ad04.slice/crio-0e17701074da26cb9e61654e613ea70bd306433c61863d1547c9d5032ecdff13 WatchSource:0}: Error finding container 0e17701074da26cb9e61654e613ea70bd306433c61863d1547c9d5032ecdff13: Status 404 returned error can't find the container with id 0e17701074da26cb9e61654e613ea70bd306433c61863d1547c9d5032ecdff13 Jul 24 22:14:40.942379 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.942348 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cdfd87_1536_465a_9409_08538c82df55.slice/crio-edb4ab61c9f6d51dba7a96a16a341b0b338f1a92c2855649aefadfce19c89508 WatchSource:0}: Error finding container edb4ab61c9f6d51dba7a96a16a341b0b338f1a92c2855649aefadfce19c89508: Status 404 returned error can't find the container with id edb4ab61c9f6d51dba7a96a16a341b0b338f1a92c2855649aefadfce19c89508 Jul 24 22:14:40.943566 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.943526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780fb566_4aca_4c9c_8c04_e794f1a4472e.slice/crio-39abf5530898f983c7976c8ec4e23564f2e9907f08a6a3a056731b6c39d56cd5 WatchSource:0}: Error finding container 39abf5530898f983c7976c8ec4e23564f2e9907f08a6a3a056731b6c39d56cd5: Status 404 returned error can't find the container with id 39abf5530898f983c7976c8ec4e23564f2e9907f08a6a3a056731b6c39d56cd5 Jul 24 22:14:40.945919 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.945901 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67a50a8_0662_4102_8bcf_1e78a529a291.slice/crio-c4d03d1f2b07bcd3ccb41d7b4d20f313095856a001621db632f19f020c7dde6a WatchSource:0}: Error finding container c4d03d1f2b07bcd3ccb41d7b4d20f313095856a001621db632f19f020c7dde6a: Status 404 returned error can't find the container with id c4d03d1f2b07bcd3ccb41d7b4d20f313095856a001621db632f19f020c7dde6a Jul 24 22:14:40.946926 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.946890 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35046547_570e_4b84_bd6d_12d666d33d7d.slice/crio-aabd2d04a6a37a753b5cf44547c3770aee672bbfe57d7965888332e4fb3796de WatchSource:0}: Error finding container aabd2d04a6a37a753b5cf44547c3770aee672bbfe57d7965888332e4fb3796de: Status 404 returned error can't find the container with id aabd2d04a6a37a753b5cf44547c3770aee672bbfe57d7965888332e4fb3796de Jul 24 22:14:40.948996 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.948971 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4689ab96_2a41_4e2e_8f6b_243f97b29cdc.slice/crio-5ba0b6d0b9b9d5b83f2a808f51b3bad6a9cdfcf06071b8b48b16e98a59a2dfb5 WatchSource:0}: Error finding container 5ba0b6d0b9b9d5b83f2a808f51b3bad6a9cdfcf06071b8b48b16e98a59a2dfb5: Status 404 returned error can't find the container with id 5ba0b6d0b9b9d5b83f2a808f51b3bad6a9cdfcf06071b8b48b16e98a59a2dfb5 Jul 24 22:14:40.950416 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.950385 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3330cf_b271_47f3_8cb8_80eea9cfaa50.slice/crio-f6ff2e0be6b32261f612058c1f22176a3dd449564a68d4b1a3ed2379a3f5dae6 WatchSource:0}: Error finding container f6ff2e0be6b32261f612058c1f22176a3dd449564a68d4b1a3ed2379a3f5dae6: Status 404 returned error can't find the container with id f6ff2e0be6b32261f612058c1f22176a3dd449564a68d4b1a3ed2379a3f5dae6 Jul 24 22:14:40.954277 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:14:40.952948 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0824b1c_cbf0_4438_afa9_b628cb1b4203.slice/crio-9b5f4dbf27bac736edf0513c3139e59ccea4765491270d2c72d597cb1910cd48 WatchSource:0}: Error finding container 9b5f4dbf27bac736edf0513c3139e59ccea4765491270d2c72d597cb1910cd48: Status 404 returned error can't find the container with id 9b5f4dbf27bac736edf0513c3139e59ccea4765491270d2c72d597cb1910cd48 Jul 24 22:14:41.393152 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.393019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:41.393315 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.393180 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:41.393315 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.393262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:14:43.393243904 +0000 UTC m=+6.060254705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:41.493950 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.493905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:41.494200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.494004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:41.494200 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494165 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:41.494325 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:43.494216992 +0000 UTC m=+6.161227816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:41.494721 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494700 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:41.494841 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494726 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:41.494841 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494741 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:41.494841 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.494792 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:14:43.494775064 +0000 UTC m=+6.161785882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:41.820799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.820737 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:09:40 +0000 UTC" deadline="2028-02-12 13:29:09.390242118 +0000 UTC" Jul 24 22:14:41.820799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.820795 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13623h14m27.56945178s" Jul 24 22:14:41.886501 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.885726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:41.886501 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.885885 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:41.886501 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.886346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:41.886501 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:41.886457 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:41.901581 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.901497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerStarted","Data":"5ba0b6d0b9b9d5b83f2a808f51b3bad6a9cdfcf06071b8b48b16e98a59a2dfb5"} Jul 24 22:14:41.903969 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.903862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"edb4ab61c9f6d51dba7a96a16a341b0b338f1a92c2855649aefadfce19c89508"} Jul 24 22:14:41.905903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.905838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8f4qr" event={"ID":"56481edd-b00d-48c6-88c7-b41cc816ad04","Type":"ContainerStarted","Data":"0e17701074da26cb9e61654e613ea70bd306433c61863d1547c9d5032ecdff13"} Jul 24 22:14:41.913267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.913239 2576 generic.go:356] "Generic (PLEG): container finished" podID="8256975cb148fd9685d00b166788932a" containerID="461ab21692b7951b1527c098dd0ad55a988e701b6777bf5587d6cffdfa5c5412" exitCode=0 Jul 24 22:14:41.913717 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.913413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" event={"ID":"8256975cb148fd9685d00b166788932a","Type":"ContainerDied","Data":"461ab21692b7951b1527c098dd0ad55a988e701b6777bf5587d6cffdfa5c5412"} Jul 24 22:14:41.935004 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.923882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjl4j" event={"ID":"0a3330cf-b271-47f3-8cb8-80eea9cfaa50","Type":"ContainerStarted","Data":"f6ff2e0be6b32261f612058c1f22176a3dd449564a68d4b1a3ed2379a3f5dae6"} Jul 24 22:14:41.939542 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.939456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6rf9" event={"ID":"35046547-570e-4b84-bd6d-12d666d33d7d","Type":"ContainerStarted","Data":"aabd2d04a6a37a753b5cf44547c3770aee672bbfe57d7965888332e4fb3796de"} Jul 24 22:14:41.941875 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.941807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" event={"ID":"c67a50a8-0662-4102-8bcf-1e78a529a291","Type":"ContainerStarted","Data":"c4d03d1f2b07bcd3ccb41d7b4d20f313095856a001621db632f19f020c7dde6a"} Jul 24 22:14:41.950008 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.949972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" event={"ID":"780fb566-4aca-4c9c-8c04-e794f1a4472e","Type":"ContainerStarted","Data":"39abf5530898f983c7976c8ec4e23564f2e9907f08a6a3a056731b6c39d56cd5"} Jul 24 22:14:41.965125 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.964991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzhmd" event={"ID":"9943b7f9-8c27-4004-ab2b-f21eeef6bf73","Type":"ContainerStarted","Data":"9c7dcb10ce5dd3d0670e60dece1b325ef669a869555283f5d68c171bdfa82cd0"} Jul 24 22:14:41.969820 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.969029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" event={"ID":"1eb745cffc865678816fc0718fd4a2e2","Type":"ContainerStarted","Data":"f669617275171955ad982d89f1aebfe03686e304515b16d54617462e88bf6a09"} Jul 24 22:14:41.974492 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:41.974415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zjk2" event={"ID":"f0824b1c-cbf0-4438-afa9-b628cb1b4203","Type":"ContainerStarted","Data":"9b5f4dbf27bac736edf0513c3139e59ccea4765491270d2c72d597cb1910cd48"} Jul 24 22:14:42.884706 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:42.884666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:42.885187 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:42.884814 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:42.999677 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:42.999634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" event={"ID":"8256975cb148fd9685d00b166788932a","Type":"ContainerStarted","Data":"bea005b231952d11a6e9a44d8b573d9fdb5ee39daddf5070a768d9953cf0fd8c"} Jul 24 22:14:43.018342 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.018263 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-132.ec2.internal" podStartSLOduration=4.018237432 podStartE2EDuration="4.018237432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:14:41.985774802 +0000 UTC m=+4.652785632" watchObservedRunningTime="2026-07-24 22:14:43.018237432 +0000 UTC m=+5.685248260" Jul 24 22:14:43.410585 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.409923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:43.410585 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.410139 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:43.410585 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.410203 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:14:47.41018504 +0000 UTC m=+10.077195845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:43.511773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.511082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:43.511773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.511172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:43.511773 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511335 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:43.511773 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511409 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:47.511388957 +0000 UTC m=+10.178399772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:43.512130 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511852 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:43.512130 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511883 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:43.512130 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511900 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:43.512130 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.511961 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:14:47.511943209 +0000 UTC m=+10.178954011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:43.887721 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.886969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:43.887721 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.887107 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:43.887721 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:43.887514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:43.887721 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:43.887647 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:44.885287 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:44.884785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:44.885287 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:44.884893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:45.885025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:45.884991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:45.885543 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:45.885134 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:45.885654 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:45.885631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:45.885785 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:45.885760 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:46.885332 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:46.884857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:46.885332 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:46.884984 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:47.445145 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:47.445059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:47.445308 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.445230 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:47.445367 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.445317 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.445295674 +0000 UTC m=+18.112306478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:47.546108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:47.546174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546271 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546287 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546304 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546316 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546331 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.546313396 +0000 UTC m=+18.213324217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:47.546696 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.546350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:14:55.546340446 +0000 UTC m=+18.213351248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:47.888369 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:47.888342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:47.888826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:47.888337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:47.888826 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.888440 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:47.888826 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:47.888507 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:48.885050 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:48.885017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:48.885225 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:48.885164 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:49.885192 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:49.884945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:49.885192 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:49.884949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:49.885192 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:49.885092 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:49.885904 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:49.885836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:50.884947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:50.884914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:50.885099 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:50.885016 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:51.887037 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:51.887009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:51.887396 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:51.887112 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:51.887396 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:51.887163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:51.887396 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:51.887246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:52.884989 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:52.884954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:52.885195 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:52.885100 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:53.887812 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:53.887774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:53.888155 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:53.887905 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:53.888155 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:53.887954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:53.888155 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:53.888040 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:54.884678 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:54.884639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:54.884948 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:54.884770 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:55.505562 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:55.505521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:55.506042 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.505698 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.506042 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.505786 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:15:11.505770554 +0000 UTC m=+34.172781353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:14:55.606953 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:55.606911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:55.607143 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:55.606989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:55.607143 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607094 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.607143 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607137 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:14:55.607262 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607152 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:14:55.607262 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:11.607139699 +0000 UTC m=+34.274150499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:14:55.607262 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607162 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:55.607262 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.607202 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:11.607190462 +0000 UTC m=+34.274201276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:14:55.884800 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:55.884761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:55.884974 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:55.884761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:55.884974 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.884905 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:55.884974 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:55.884953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:56.884050 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:56.884013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:56.884508 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:56.884147 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:57.885231 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:57.885191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:57.885717 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:57.885316 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:14:57.885717 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:57.885368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:57.885717 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:57.885487 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:58.884693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:58.884654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:14:58.884879 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:58.884772 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:14:59.884340 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:59.884309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:14:59.884794 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:14:59.884309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:14:59.884794 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:59.884451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:14:59.884794 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:14:59.884485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:00.884592 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:00.884400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:00.885353 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:00.884666 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:01.051697 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.051656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" event={"ID":"c67a50a8-0662-4102-8bcf-1e78a529a291","Type":"ContainerStarted","Data":"53688a2e7ed1b1d4e1af5d346f4d49c46dd3e51a89b9a06f83e5adebea08135d"} Jul 24 22:15:01.053142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.053119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" event={"ID":"780fb566-4aca-4c9c-8c04-e794f1a4472e","Type":"ContainerStarted","Data":"fdbe41d92ce6cd701304d77ff1f05f0cec75a2bc5faa11b220e9806c4fdd886c"} Jul 24 22:15:01.054306 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.054285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zzhmd" event={"ID":"9943b7f9-8c27-4004-ab2b-f21eeef6bf73","Type":"ContainerStarted","Data":"d3b33553c9c3dac9f379f9bb4eefb0fc49964418c87b65019f9714b3007da62c"} Jul 24 22:15:01.055737 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.055702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zjk2" event={"ID":"f0824b1c-cbf0-4438-afa9-b628cb1b4203","Type":"ContainerStarted","Data":"e4a66a56ee4831934b36d20cf42fc7dea41ffe2a5e2b21d5ee9ed073594293c7"} Jul 24 22:15:01.057065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.057035 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="cf4b3ae5454f99ae700511b304321f276dc4faef33ea0e695cb730cded27d74c" exitCode=0 Jul 24 22:15:01.057147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.057116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"cf4b3ae5454f99ae700511b304321f276dc4faef33ea0e695cb730cded27d74c"} Jul 24 22:15:01.060651 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"fea21de034ba893aea317a5e8419ccb4b1637f3749f4fabda3030e4a073b7502"} Jul 24 22:15:01.060728 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"191d909f777ae04ef29c55f576670bade12b3aaeea0b26a9eec37caa4ba7a24b"} Jul 24 22:15:01.060728 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"6cd0c528035e15872f41ea6c1e6270802e2079bdac1c1d662d77f4a995eef796"} Jul 24 22:15:01.060814 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"8e8ca9c3827349aff9d4a1cd00e6044b8a48c268998a5d0c36628b694abf5a62"} Jul 24 22:15:01.060814 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"841a4c8f46ecd636aa631ed640226f2361901cf0a94c4df4f929887f3ca8cbbb"} Jul 24 22:15:01.060814 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.060814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"9be3d29b5c0076297c7cab403c0a0019e595f776351730f8f4d8b7817a43f7e1"} Jul 24 22:15:01.062074 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.062044 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8f4qr" event={"ID":"56481edd-b00d-48c6-88c7-b41cc816ad04","Type":"ContainerStarted","Data":"d8918ccce76b87cccecc12a421f0fb967fb7f8b1768dbac40a121d06ca03427b"} Jul 24 22:15:01.063369 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.063349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjl4j" event={"ID":"0a3330cf-b271-47f3-8cb8-80eea9cfaa50","Type":"ContainerStarted","Data":"c29de258acd15dfd02996fe08265be45bcd37e4fd458f12520e54b3f613866dc"} Jul 24 22:15:01.069932 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.069893 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l2gz2" podStartSLOduration=4.109065408 podStartE2EDuration="23.06988004s" totalImagesPullingTime="18.960814632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:01.069312202 +0000 UTC m=+23.736323023" watchObservedRunningTime="2026-07-24 22:15:01.06988004 +0000 UTC m=+23.736890861" Jul 24 22:15:01.070031 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.069981 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-132.ec2.internal" podStartSLOduration=22.069977253 podStartE2EDuration="22.069977253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:14:43.018765833 +0000 UTC m=+5.685776659" watchObservedRunningTime="2026-07-24 22:15:01.069977253 +0000 UTC m=+23.736988076" Jul 24 22:15:01.100783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.100736 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6zjk2" podStartSLOduration=4.152840677 podStartE2EDuration="23.100718985s" totalImagesPullingTime="18.947878308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:01.100476821 +0000 UTC m=+23.767487664" watchObservedRunningTime="2026-07-24 22:15:01.100718985 +0000 UTC m=+23.767729807" Jul 24 22:15:01.100945 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.100806 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bjl4j" podStartSLOduration=4.14941198 podStartE2EDuration="23.100801739s" totalImagesPullingTime="18.951389759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:01.0852715 +0000 UTC m=+23.752282321" watchObservedRunningTime="2026-07-24 22:15:01.100801739 +0000 UTC m=+23.767812561" Jul 24 22:15:01.163986 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.163752 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zzhmd" podStartSLOduration=8.153770559 podStartE2EDuration="23.163734946s" totalImagesPullingTime="15.009964387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:01.124902479 +0000 UTC m=+23.791913301" watchObservedRunningTime="2026-07-24 22:15:01.163734946 +0000 UTC m=+23.830745828" Jul 24 22:15:01.207598 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.207528 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8f4qr" podStartSLOduration=4.18517408 podStartE2EDuration="23.207508918s" totalImagesPullingTime="19.022334838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:01.163585421 +0000 UTC m=+23.830596242" watchObservedRunningTime="2026-07-24 22:15:01.207508918 +0000 UTC m=+23.874519739" Jul 24 22:15:01.227202 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.227171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:15:01.227967 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.227952 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:15:01.393426 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.393400 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 22:15:01.835061 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.834861 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T22:15:01.393417714Z","UUID":"e356f3c4-e341-49f3-b296-e95c0c6b73ee","Handler":null,"Name":"","Endpoint":""} Jul 24 22:15:01.836738 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.836718 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 22:15:01.836864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.836759 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 22:15:01.884485 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.884452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:01.884663 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:01.884597 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:01.885206 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:01.885182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:01.885529 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:01.885505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:02.069375 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.069324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6rf9" event={"ID":"35046547-570e-4b84-bd6d-12d666d33d7d","Type":"ContainerStarted","Data":"e7f72e229c2221fbd80c4081bc0c24991f0be9bb60a9ab9a93687f1676fb5e50"} Jul 24 22:15:02.072181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.072144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" event={"ID":"780fb566-4aca-4c9c-8c04-e794f1a4472e","Type":"ContainerStarted","Data":"8f60f5eb6acc91bb5c5360fe93bde4d6ffbe8759e9f3c6b82481245c46a1af05"} Jul 24 22:15:02.073016 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.072967 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:15:02.073444 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.073424 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zzhmd" Jul 24 22:15:02.120988 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.120928 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x6rf9" podStartSLOduration=5.163935127 podStartE2EDuration="24.120903671s" totalImagesPullingTime="18.956968544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:02.097854136 +0000 UTC m=+24.764864961" watchObservedRunningTime="2026-07-24 22:15:02.120903671 +0000 UTC m=+24.787914494" Jul 24 22:15:02.884409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:02.884061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:02.884409 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:02.884180 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:03.077305 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:03.077264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" event={"ID":"780fb566-4aca-4c9c-8c04-e794f1a4472e","Type":"ContainerStarted","Data":"2ee2d28e75430fa3f60aecb0bd62df3b20d4aa6dd44bd15f6f601ac84400354f"} Jul 24 22:15:03.081253 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:03.081218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"1a5e0f8e99f4760d39057428440e8b6d6f06932620b8868e225440755b4ab816"} Jul 24 22:15:03.884805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:03.884772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:03.885064 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:03.884814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:03.885064 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:03.884922 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:03.885064 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:03.885030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:04.884690 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:04.884653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:04.885120 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:04.884758 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:05.884419 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:05.884240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:05.884666 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:05.884301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:05.884666 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:05.884493 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:05.884666 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:05.884607 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:06.090741 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.090706 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="87f486e1664370199c1127b89aa9848957cdef295507f087bb848d546298bbe2" exitCode=0 Jul 24 22:15:06.090909 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.090795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"87f486e1664370199c1127b89aa9848957cdef295507f087bb848d546298bbe2"} Jul 24 22:15:06.094546 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.094517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" event={"ID":"46cdfd87-1536-465a-9409-08538c82df55","Type":"ContainerStarted","Data":"9498d91ed26190090aa08c98cddf60ea07520c0fe507279a5c6d5eec07b39633"} Jul 24 22:15:06.094843 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.094825 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:06.094914 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.094884 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:06.111077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.111049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:06.115970 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.115924 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g49bf" podStartSLOduration=6.699370079 podStartE2EDuration="28.11590769s" totalImagesPullingTime="21.416537611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:03.103460591 +0000 UTC m=+25.770471413" watchObservedRunningTime="2026-07-24 22:15:06.11590769 +0000 UTC m=+28.782918578" Jul 24 22:15:06.141973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.141889 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" podStartSLOduration=9.118507433 podStartE2EDuration="28.141872929s" totalImagesPullingTime="19.023365496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:06.141546745 +0000 UTC m=+28.808557567" watchObservedRunningTime="2026-07-24 22:15:06.141872929 +0000 UTC m=+28.808883751" Jul 24 22:15:06.884472 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:06.884444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:06.884700 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:06.884586 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:07.101240 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.101073 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="7d74fdb4382ea28429e6d32a9d20fe6d484aca8a1a3a65c3f1f628975fe257f3" exitCode=0 Jul 24 22:15:07.101797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.101158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"7d74fdb4382ea28429e6d32a9d20fe6d484aca8a1a3a65c3f1f628975fe257f3"} Jul 24 22:15:07.101797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.101759 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:07.118284 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.118260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:07.330629 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.330597 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vcvw9"] Jul 24 22:15:07.330815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.330732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:07.330882 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:07.330821 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:07.337212 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.337178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vhmh"] Jul 24 22:15:07.337355 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.337297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:07.337424 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:07.337405 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:07.349669 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.349633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vpxvh"] Jul 24 22:15:07.349825 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:07.349767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:07.349895 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:07.349874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:08.884667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:08.884637 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:08.884667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:08.884668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:08.885075 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:08.884731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:08.885075 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:08.884745 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:08.885075 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:08.884839 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:08.885075 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:08.884938 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:09.109097 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:09.109065 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="8995cffac1b008d09ed9df81723ef1c877a5c66a7e578877e6e576276024e8e2" exitCode=0 Jul 24 22:15:09.109251 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:09.109142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"8995cffac1b008d09ed9df81723ef1c877a5c66a7e578877e6e576276024e8e2"} Jul 24 22:15:10.884931 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:10.884895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:10.884931 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:10.884924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:10.885485 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:10.884902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:10.885485 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:10.885027 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:10.885485 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:10.885090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:10.885485 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:10.885185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:11.520749 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:11.520715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:11.520932 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.520886 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:11.520992 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.520966 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:15:43.520948109 +0000 UTC m=+66.187958926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:15:11.621632 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:11.621588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:11.621824 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:11.621685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:11.621824 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621792 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:15:11.621824 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621800 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:11.621824 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621815 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:15:11.621824 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621827 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7mhrk for pod openshift-network-diagnostics/network-check-target-vcvw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:11.622062 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret podName:dc3a409d-d52b-4761-901f-a7a4c6ca7ca4 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:43.621853841 +0000 UTC m=+66.288864657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret") pod "global-pull-secret-syncer-vpxvh" (UID: "dc3a409d-d52b-4761-901f-a7a4c6ca7ca4") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:15:11.622062 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:11.621892 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk podName:c69a5fbd-cb1c-470d-b6ed-b85ead07623d nodeName:}" failed. No retries permitted until 2026-07-24 22:15:43.621882958 +0000 UTC m=+66.288893761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7mhrk" (UniqueName: "kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk") pod "network-check-target-vcvw9" (UID: "c69a5fbd-cb1c-470d-b6ed-b85ead07623d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:15:12.884318 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:12.884277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:12.884318 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:12.884304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:12.884873 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:12.884289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:12.884873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:12.884429 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vhmh" podUID="f1654044-8118-4cc6-8743-397a4adab49f" Jul 24 22:15:12.884873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:12.884588 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcvw9" podUID="c69a5fbd-cb1c-470d-b6ed-b85ead07623d" Jul 24 22:15:12.884873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:12.884700 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vpxvh" podUID="dc3a409d-d52b-4761-901f-a7a4c6ca7ca4" Jul 24 22:15:13.004830 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.004757 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-132.ec2.internal" event="NodeReady" Jul 24 22:15:13.004973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.004926 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 22:15:13.040073 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.040041 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:15:13.042326 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.042306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.044182 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.044164 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 22:15:13.044440 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.044422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 22:15:13.044525 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.044439 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 22:15:13.044525 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.044458 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sr77c\"" Jul 24 22:15:13.050798 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.050589 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 22:15:13.054925 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.054899 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gmhgc"] Jul 24 22:15:13.056622 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.056603 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8c4mn"] Jul 24 22:15:13.056718 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.056703 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.058419 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.058403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.059846 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.059823 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 22:15:13.059950 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.059933 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wnkgk\"" Jul 24 22:15:13.060126 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060109 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 22:15:13.060236 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 22:15:13.060360 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060275 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 22:15:13.060360 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060308 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 22:15:13.060539 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060525 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:15:13.060647 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.060529 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ck9v8\"" Jul 24 22:15:13.069409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.069386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8c4mn"] Jul 24 22:15:13.072079 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.072058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gmhgc"] Jul 24 22:15:13.122105 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.122057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerStarted","Data":"096d18c8d8d793594abcbc7a99fbe231fc677703b2ca6fbf95cb7eff4280a685"} Jul 24 22:15:13.134768 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.134768 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.134768 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vkv\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.134927 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.134927 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08c4a51f-148d-4a9f-a9cd-dee108d95b40-config-volume\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.134927 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.134927 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.135051 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.135051 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.134988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.135051 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.135014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cb2a34-e342-4afa-8915-ac6efad06772-cert\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.135051 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.135037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.135180 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.135054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c4a51f-148d-4a9f-a9cd-dee108d95b40-tmp-dir\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.135180 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.135070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8xl\" (UniqueName: \"kubernetes.io/projected/e8cb2a34-e342-4afa-8915-ac6efad06772-kube-api-access-bm8xl\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.135180 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.135085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5r7\" (UniqueName: \"kubernetes.io/projected/08c4a51f-148d-4a9f-a9cd-dee108d95b40-kube-api-access-kj5r7\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.235839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.235839 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cb2a34-e342-4afa-8915-ac6efad06772-cert\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c4a51f-148d-4a9f-a9cd-dee108d95b40-tmp-dir\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8xl\" (UniqueName: \"kubernetes.io/projected/e8cb2a34-e342-4afa-8915-ac6efad06772-kube-api-access-bm8xl\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.235966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5r7\" (UniqueName: \"kubernetes.io/projected/08c4a51f-148d-4a9f-a9cd-dee108d95b40-kube-api-access-kj5r7\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.236054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236328 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236328 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vkv\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236328 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.236166 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:13.236328 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.236183 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:13.236328 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.236249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:13.736231473 +0000 UTC m=+36.403242302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:13.236597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c4a51f-148d-4a9f-a9cd-dee108d95b40-tmp-dir\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.236597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08c4a51f-148d-4a9f-a9cd-dee108d95b40-config-volume\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.236597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.236820 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.236616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.236820 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.236712 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:13.236820 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.236757 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:13.736740485 +0000 UTC m=+36.403751286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:13.237071 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.237050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.237164 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.237147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08c4a51f-148d-4a9f-a9cd-dee108d95b40-config-volume\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.237211 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.237174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.240588 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.240543 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.240588 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.240578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cb2a34-e342-4afa-8915-ac6efad06772-cert\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.240787 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.240579 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.243711 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.243684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zzhmd_9943b7f9-8c27-4004-ab2b-f21eeef6bf73/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:13.249087 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.249062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vkv\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.253657 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.253634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8xl\" (UniqueName: \"kubernetes.io/projected/e8cb2a34-e342-4afa-8915-ac6efad06772-kube-api-access-bm8xl\") pod \"ingress-canary-8c4mn\" (UID: \"e8cb2a34-e342-4afa-8915-ac6efad06772\") " pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.253842 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.253827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.253893 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.253873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5r7\" (UniqueName: \"kubernetes.io/projected/08c4a51f-148d-4a9f-a9cd-dee108d95b40-kube-api-access-kj5r7\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.389214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.389129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8c4mn" Jul 24 22:15:13.524309 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.524277 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8c4mn"] Jul 24 22:15:13.530221 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:13.530187 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cb2a34_e342_4afa_8915_ac6efad06772.slice/crio-0ffe98db5d6dba506e74d14ac5ed8a2d98f2fca6b6a69381a87509a45f8b5309 WatchSource:0}: Error finding container 0ffe98db5d6dba506e74d14ac5ed8a2d98f2fca6b6a69381a87509a45f8b5309: Status 404 returned error can't find the container with id 0ffe98db5d6dba506e74d14ac5ed8a2d98f2fca6b6a69381a87509a45f8b5309 Jul 24 22:15:13.740831 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.740728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:13.740989 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:13.740868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:13.740989 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.740886 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:13.740989 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.740949 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:14.740929967 +0000 UTC m=+37.407940783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:13.740989 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.740983 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:13.741117 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.740994 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:13.741117 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:13.741036 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:14.741024979 +0000 UTC m=+37.408035779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:14.125765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.125723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8c4mn" event={"ID":"e8cb2a34-e342-4afa-8915-ac6efad06772","Type":"ContainerStarted","Data":"0ffe98db5d6dba506e74d14ac5ed8a2d98f2fca6b6a69381a87509a45f8b5309"} Jul 24 22:15:14.128818 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.128783 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="096d18c8d8d793594abcbc7a99fbe231fc677703b2ca6fbf95cb7eff4280a685" exitCode=0 Jul 24 22:15:14.128921 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.128830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"096d18c8d8d793594abcbc7a99fbe231fc677703b2ca6fbf95cb7eff4280a685"} Jul 24 22:15:14.359039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.358844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zzhmd_9943b7f9-8c27-4004-ab2b-f21eeef6bf73/konnectivity-agent/0.log" limit=1024 Jul 24 22:15:14.749719 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.749683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:14.749930 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.749742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:14.749930 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:14.749846 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:14.749930 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:14.749854 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:14.749930 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:14.749877 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:14.749930 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:14.749918 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:16.749899728 +0000 UTC m=+39.416910529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:14.750133 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:14.749937 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:16.749927778 +0000 UTC m=+39.416938579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:14.884158 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.884124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:14.884343 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.884132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:14.884343 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.884258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:14.886248 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.886225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:15:14.886248 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.886239 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:15:14.886394 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.886225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rtdx2\"" Jul 24 22:15:14.887142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.886941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:15:14.887142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.886940 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:15:14.887142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:14.887039 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-767d7\"" Jul 24 22:15:15.135241 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:15.135204 2576 generic.go:356] "Generic (PLEG): container finished" podID="4689ab96-2a41-4e2e-8f6b-243f97b29cdc" containerID="a7f75a72816da9150a31dd7ec98fd7d1798273183828583b5eb227d0fc0c5a65" exitCode=0 Jul 24 22:15:15.135665 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:15.135272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerDied","Data":"a7f75a72816da9150a31dd7ec98fd7d1798273183828583b5eb227d0fc0c5a65"} Jul 24 22:15:16.139137 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.139104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8c4mn" event={"ID":"e8cb2a34-e342-4afa-8915-ac6efad06772","Type":"ContainerStarted","Data":"3fede6b95bdf89855244a376214b12d611e44c7829d1dd5853fa3005c61539c1"} Jul 24 22:15:16.142907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.142869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" event={"ID":"4689ab96-2a41-4e2e-8f6b-243f97b29cdc","Type":"ContainerStarted","Data":"f89d677482516f4ab831c70b6467f7b423402219f8577c1aafa7764a4481c21f"} Jul 24 22:15:16.156131 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.156073 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8c4mn" podStartSLOduration=0.822676685 podStartE2EDuration="3.1560537s" totalImagesPullingTime="2.333377015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:16.155707275 +0000 UTC m=+38.822718096" watchObservedRunningTime="2026-07-24 22:15:16.1560537 +0000 UTC m=+38.823064523" Jul 24 22:15:16.175818 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.175727 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b8kmm" podStartSLOduration=9.605976222 podStartE2EDuration="38.175708275s" totalImagesPullingTime="28.569732053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:16.174838529 +0000 UTC m=+38.841849352" watchObservedRunningTime="2026-07-24 22:15:16.175708275 +0000 UTC m=+38.842719095" Jul 24 22:15:16.766410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.766370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:16.766599 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:16.766424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:16.766599 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:16.766517 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:16.766599 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:16.766532 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:16.766599 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:16.766534 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:16.766819 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:16.766611 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:20.766594765 +0000 UTC m=+43.433605564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:16.766819 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:16.766626 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:20.766619861 +0000 UTC m=+43.433630661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:20.800584 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:20.800519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:20.800610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:20.800697 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:20.800727 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:20.800740 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:20.800796 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:28.80077651 +0000 UTC m=+51.467787309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:20.800993 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:20.800810 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:28.800804078 +0000 UTC m=+51.467814877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:28.861398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:28.861356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:28.861398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:28.861409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:28.861956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:28.861498 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:28.861956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:28.861519 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:28.861956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:28.861519 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:28.861956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:28.861586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:44.861571712 +0000 UTC m=+67.528582524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:28.861956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:28.861598 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:15:44.861592464 +0000 UTC m=+67.528603264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:39.119743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:39.119715 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4pkmq" Jul 24 22:15:43.569267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.569228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:15:43.571373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.571358 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:15:43.580153 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:43.580137 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 22:15:43.580200 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:43.580189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs podName:f1654044-8118-4cc6-8743-397a4adab49f nodeName:}" failed. No retries permitted until 2026-07-24 22:16:47.580174208 +0000 UTC m=+130.247185022 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs") pod "network-metrics-daemon-7vhmh" (UID: "f1654044-8118-4cc6-8743-397a4adab49f") : secret "metrics-daemon-secret" not found Jul 24 22:15:43.669807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.669770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:43.669872 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.669846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:43.671937 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.671921 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:15:43.672006 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.671960 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:15:43.682238 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.682214 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:15:43.684105 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.684080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc3a409d-d52b-4761-901f-a7a4c6ca7ca4-original-pull-secret\") pod \"global-pull-secret-syncer-vpxvh\" (UID: \"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4\") " pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:43.693264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.693242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhrk\" (UniqueName: \"kubernetes.io/projected/c69a5fbd-cb1c-470d-b6ed-b85ead07623d-kube-api-access-7mhrk\") pod \"network-check-target-vcvw9\" (UID: \"c69a5fbd-cb1c-470d-b6ed-b85ead07623d\") " pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:43.696849 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.696834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vpxvh" Jul 24 22:15:43.708690 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.708669 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rtdx2\"" Jul 24 22:15:43.717330 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.717310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:43.842229 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.842151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vpxvh"] Jul 24 22:15:43.846285 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:43.846247 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3a409d_d52b_4761_901f_a7a4c6ca7ca4.slice/crio-350e4411bec7771a7a1244ccf712a2515469a2e42fc886eb492cb70cfb7958eb WatchSource:0}: Error finding container 350e4411bec7771a7a1244ccf712a2515469a2e42fc886eb492cb70cfb7958eb: Status 404 returned error can't find the container with id 350e4411bec7771a7a1244ccf712a2515469a2e42fc886eb492cb70cfb7958eb Jul 24 22:15:43.861206 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:43.861174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vcvw9"] Jul 24 22:15:43.864324 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:43.864301 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69a5fbd_cb1c_470d_b6ed_b85ead07623d.slice/crio-61b4c567c5c7747209b71f45042c0efc4cb71d5e015679baf246d86b2b1f7d4e WatchSource:0}: Error finding container 61b4c567c5c7747209b71f45042c0efc4cb71d5e015679baf246d86b2b1f7d4e: Status 404 returned error can't find the container with id 61b4c567c5c7747209b71f45042c0efc4cb71d5e015679baf246d86b2b1f7d4e Jul 24 22:15:44.208703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:44.208666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vcvw9" event={"ID":"c69a5fbd-cb1c-470d-b6ed-b85ead07623d","Type":"ContainerStarted","Data":"61b4c567c5c7747209b71f45042c0efc4cb71d5e015679baf246d86b2b1f7d4e"} Jul 24 22:15:44.209586 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:44.209544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vpxvh" event={"ID":"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4","Type":"ContainerStarted","Data":"350e4411bec7771a7a1244ccf712a2515469a2e42fc886eb492cb70cfb7958eb"} Jul 24 22:15:44.878474 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:44.878441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:44.878495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:44.878630 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:44.878653 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56787b77dc-8fbkn: secret "image-registry-tls" not found Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:44.878673 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:44.878738 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls podName:08c4a51f-148d-4a9f-a9cd-dee108d95b40 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:16.878716427 +0000 UTC m=+99.545727230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls") pod "dns-default-gmhgc" (UID: "08c4a51f-148d-4a9f-a9cd-dee108d95b40") : secret "dns-default-metrics-tls" not found Jul 24 22:15:44.878971 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:44.878756 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls podName:fc39537f-3022-4277-aabd-60bff510e2bb nodeName:}" failed. No retries permitted until 2026-07-24 22:16:16.878747024 +0000 UTC m=+99.545757830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls") pod "image-registry-56787b77dc-8fbkn" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb") : secret "image-registry-tls" not found Jul 24 22:15:45.665425 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.665384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f"] Jul 24 22:15:45.672111 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.672087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.675039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.674680 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 22:15:45.675039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.674892 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 22:15:45.675039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.674908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-97dfg\"" Jul 24 22:15:45.675275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.675170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 22:15:45.675275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.675254 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:15:45.678535 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.678505 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f"] Jul 24 22:15:45.685872 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.685847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e448e3b7-5558-42e8-bb21-78cb00b9898c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.685991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.685884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224q8\" (UniqueName: \"kubernetes.io/projected/e448e3b7-5558-42e8-bb21-78cb00b9898c-kube-api-access-224q8\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.685991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.685925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e448e3b7-5558-42e8-bb21-78cb00b9898c-config\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.763329 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.763291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh"] Jul 24 22:15:45.780782 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.780759 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr"] Jul 24 22:15:45.780931 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.780870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" Jul 24 22:15:45.783292 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.783271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tqcmd\"" Jul 24 22:15:45.786572 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.786478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb8z\" (UniqueName: \"kubernetes.io/projected/df9f56d5-ac90-48d5-936e-e25565403c5a-kube-api-access-qlb8z\") pod \"network-check-source-7d88cb55d8-wxhgh\" (UID: \"df9f56d5-ac90-48d5-936e-e25565403c5a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" Jul 24 22:15:45.786572 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.786526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e448e3b7-5558-42e8-bb21-78cb00b9898c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.786572 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.786568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224q8\" (UniqueName: \"kubernetes.io/projected/e448e3b7-5558-42e8-bb21-78cb00b9898c-kube-api-access-224q8\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.786841 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.786605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e448e3b7-5558-42e8-bb21-78cb00b9898c-config\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.787374 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.787307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e448e3b7-5558-42e8-bb21-78cb00b9898c-config\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.789311 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.789270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e448e3b7-5558-42e8-bb21-78cb00b9898c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.795767 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.795746 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-224q8\" (UniqueName: \"kubernetes.io/projected/e448e3b7-5558-42e8-bb21-78cb00b9898c-kube-api-access-224q8\") pod \"service-ca-operator-5d6c75f84f-bl59f\" (UID: \"e448e3b7-5558-42e8-bb21-78cb00b9898c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.811295 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.811274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh"] Jul 24 22:15:45.811409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.811309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr"] Jul 24 22:15:45.811409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.811385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" Jul 24 22:15:45.813527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.813467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:15:45.813527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.813490 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 22:15:45.813993 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.813813 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kfvt6\"" Jul 24 22:15:45.887133 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.887096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbkh\" (UniqueName: \"kubernetes.io/projected/8a40dd43-2cff-4f53-ad6e-d8deefb954ba-kube-api-access-npbkh\") pod \"volume-data-source-validator-549c6d54cf-cmpjr\" (UID: \"8a40dd43-2cff-4f53-ad6e-d8deefb954ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" Jul 24 22:15:45.887593 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.887175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb8z\" (UniqueName: \"kubernetes.io/projected/df9f56d5-ac90-48d5-936e-e25565403c5a-kube-api-access-qlb8z\") pod \"network-check-source-7d88cb55d8-wxhgh\" (UID: \"df9f56d5-ac90-48d5-936e-e25565403c5a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" Jul 24 22:15:45.895661 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.895632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb8z\" (UniqueName: \"kubernetes.io/projected/df9f56d5-ac90-48d5-936e-e25565403c5a-kube-api-access-qlb8z\") pod \"network-check-source-7d88cb55d8-wxhgh\" (UID: \"df9f56d5-ac90-48d5-936e-e25565403c5a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" Jul 24 22:15:45.984411 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.984330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" Jul 24 22:15:45.987715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.987690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbkh\" (UniqueName: \"kubernetes.io/projected/8a40dd43-2cff-4f53-ad6e-d8deefb954ba-kube-api-access-npbkh\") pod \"volume-data-source-validator-549c6d54cf-cmpjr\" (UID: \"8a40dd43-2cff-4f53-ad6e-d8deefb954ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" Jul 24 22:15:45.997387 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:45.997355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbkh\" (UniqueName: \"kubernetes.io/projected/8a40dd43-2cff-4f53-ad6e-d8deefb954ba-kube-api-access-npbkh\") pod \"volume-data-source-validator-549c6d54cf-cmpjr\" (UID: \"8a40dd43-2cff-4f53-ad6e-d8deefb954ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" Jul 24 22:15:46.104344 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:46.104304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" Jul 24 22:15:46.121251 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:46.121223 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" Jul 24 22:15:48.472237 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:48.472209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f"] Jul 24 22:15:48.478044 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:48.478011 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode448e3b7_5558_42e8_bb21_78cb00b9898c.slice/crio-a3633fe9843e86b31a159d094b9f7c2f57209fad7963ad1508386e128400edf3 WatchSource:0}: Error finding container a3633fe9843e86b31a159d094b9f7c2f57209fad7963ad1508386e128400edf3: Status 404 returned error can't find the container with id a3633fe9843e86b31a159d094b9f7c2f57209fad7963ad1508386e128400edf3 Jul 24 22:15:48.689714 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:48.689678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr"] Jul 24 22:15:48.690441 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:48.690349 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh"] Jul 24 22:15:48.692670 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:48.692648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a40dd43_2cff_4f53_ad6e_d8deefb954ba.slice/crio-efd1c9807924616aaec66f248f416b19578bd1102ed90423ef27efb06d9b4944 WatchSource:0}: Error finding container efd1c9807924616aaec66f248f416b19578bd1102ed90423ef27efb06d9b4944: Status 404 returned error can't find the container with id efd1c9807924616aaec66f248f416b19578bd1102ed90423ef27efb06d9b4944 Jul 24 22:15:48.693137 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:48.693115 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f56d5_ac90_48d5_936e_e25565403c5a.slice/crio-b24c6f6c4d808f3f2a50ceb2f53272bd1ae902310c9ea5792be3a3d356187ebb WatchSource:0}: Error finding container b24c6f6c4d808f3f2a50ceb2f53272bd1ae902310c9ea5792be3a3d356187ebb: Status 404 returned error can't find the container with id b24c6f6c4d808f3f2a50ceb2f53272bd1ae902310c9ea5792be3a3d356187ebb Jul 24 22:15:49.070921 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.070838 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lzzjg"] Jul 24 22:15:49.073372 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.073337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.076651 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.076620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 22:15:49.077387 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.077362 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 22:15:49.077707 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.077687 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 22:15:49.077939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.077893 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nftg\"" Jul 24 22:15:49.078013 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.077947 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 22:15:49.107917 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.107835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzzjg"] Jul 24 22:15:49.114325 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.114294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b983e892-a4d5-4ed3-9e28-4bce97b48431-crio-socket\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.114466 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.114339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8plf5\" (UniqueName: \"kubernetes.io/projected/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-api-access-8plf5\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.114466 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.114434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.114600 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.114464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.114600 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.114512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b983e892-a4d5-4ed3-9e28-4bce97b48431-data-volume\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.215914 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.215879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b983e892-a4d5-4ed3-9e28-4bce97b48431-crio-socket\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.215927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8plf5\" (UniqueName: \"kubernetes.io/projected/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-api-access-8plf5\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.216029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.216045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b983e892-a4d5-4ed3-9e28-4bce97b48431-crio-socket\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.216055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.216094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b983e892-a4d5-4ed3-9e28-4bce97b48431-data-volume\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.216378 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.216126 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:49.216378 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.216184 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.716168637 +0000 UTC m=+72.383179438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : configmap "kube-rbac-proxy" not found Jul 24 22:15:49.216378 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.216344 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.216576 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.216392 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:49.71637691 +0000 UTC m=+72.383387723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.216633 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.216601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b983e892-a4d5-4ed3-9e28-4bce97b48431-data-volume\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.226704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.226638 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" event={"ID":"e448e3b7-5558-42e8-bb21-78cb00b9898c","Type":"ContainerStarted","Data":"a3633fe9843e86b31a159d094b9f7c2f57209fad7963ad1508386e128400edf3"} Jul 24 22:15:49.227127 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.227105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8plf5\" (UniqueName: \"kubernetes.io/projected/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-api-access-8plf5\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.228762 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.228738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vcvw9" event={"ID":"c69a5fbd-cb1c-470d-b6ed-b85ead07623d","Type":"ContainerStarted","Data":"b6fd92b3db1274ce94210f2abba3cf53c25887fceb68a279e301f606d578e150"} Jul 24 22:15:49.228883 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.228850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:15:49.230682 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.230336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vpxvh" event={"ID":"dc3a409d-d52b-4761-901f-a7a4c6ca7ca4","Type":"ContainerStarted","Data":"9bb64db06254e3e3a0b4caa09d3e9742a5949c97f18356720ccd854bdc9b3bd8"} Jul 24 22:15:49.232159 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.232138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" event={"ID":"df9f56d5-ac90-48d5-936e-e25565403c5a","Type":"ContainerStarted","Data":"8b6113a62eb00aab2e0bfab6b20cd2b60761217b2626a161f6589b1e453dbe55"} Jul 24 22:15:49.232275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.232175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" event={"ID":"df9f56d5-ac90-48d5-936e-e25565403c5a","Type":"ContainerStarted","Data":"b24c6f6c4d808f3f2a50ceb2f53272bd1ae902310c9ea5792be3a3d356187ebb"} Jul 24 22:15:49.233353 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.233332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" event={"ID":"8a40dd43-2cff-4f53-ad6e-d8deefb954ba","Type":"ContainerStarted","Data":"efd1c9807924616aaec66f248f416b19578bd1102ed90423ef27efb06d9b4944"} Jul 24 22:15:49.246297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.246248 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vcvw9" podStartSLOduration=66.795460383 podStartE2EDuration="1m11.246234973s" totalImagesPullingTime="4.45077459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:49.244937667 +0000 UTC m=+71.911948488" watchObservedRunningTime="2026-07-24 22:15:49.246234973 +0000 UTC m=+71.913245793" Jul 24 22:15:49.258914 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.258850 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vpxvh" podStartSLOduration=66.772812301 podStartE2EDuration="1m11.258832754s" totalImagesPullingTime="4.486020453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:49.256981416 +0000 UTC m=+71.923992235" watchObservedRunningTime="2026-07-24 22:15:49.258832754 +0000 UTC m=+71.925843576" Jul 24 22:15:49.270698 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.269708 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wxhgh" podStartSLOduration=4.269693521 podStartE2EDuration="4.269693521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:49.269150918 +0000 UTC m=+71.936161743" watchObservedRunningTime="2026-07-24 22:15:49.269693521 +0000 UTC m=+71.936704342" Jul 24 22:15:49.721361 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.721327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.721361 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:49.721374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:49.721873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.721490 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:49.721873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.721505 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.721873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.721577 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:50.721543811 +0000 UTC m=+73.388554611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:49.721873 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:49.721594 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:50.72158605 +0000 UTC m=+73.388596850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : configmap "kube-rbac-proxy" not found Jul 24 22:15:50.730573 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:50.730466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:50.730573 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:50.730523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:50.731012 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:50.730608 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:50.731012 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:50.730674 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:52.730657314 +0000 UTC m=+75.397668116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : configmap "kube-rbac-proxy" not found Jul 24 22:15:50.731012 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:50.730697 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:50.731012 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:50.730774 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:52.730754207 +0000 UTC m=+75.397765013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:51.051522 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:51.051444 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zjk2_f0824b1c-cbf0-4438-afa9-b628cb1b4203/dns-node-resolver/0.log" Jul 24 22:15:51.242128 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:51.242088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" event={"ID":"8a40dd43-2cff-4f53-ad6e-d8deefb954ba","Type":"ContainerStarted","Data":"e72acca8444af203bebf4b1be6b05d90cbf9b24d11dfd83b5984072c962ac3d4"} Jul 24 22:15:51.243580 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:51.243539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" event={"ID":"e448e3b7-5558-42e8-bb21-78cb00b9898c","Type":"ContainerStarted","Data":"854683974a5d884f5f1bff0272de6e6f509138a5002da071f9f0fe74e9852289"} Jul 24 22:15:51.256901 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:51.256850 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cmpjr" podStartSLOduration=4.478157459 podStartE2EDuration="6.256833593s" totalImagesPullingTime="1.778676134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:51.255852369 +0000 UTC m=+73.922863190" watchObservedRunningTime="2026-07-24 22:15:51.256833593 +0000 UTC m=+73.923844392" Jul 24 22:15:51.269682 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:51.269629 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" podStartSLOduration=4.271918659 podStartE2EDuration="6.269612261s" totalImagesPullingTime="1.997693602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:51.268922152 +0000 UTC m=+73.935932973" watchObservedRunningTime="2026-07-24 22:15:51.269612261 +0000 UTC m=+73.936623132" Jul 24 22:15:52.052606 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:52.052579 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjl4j_0a3330cf-b271-47f3-8cb8-80eea9cfaa50/node-ca/0.log" Jul 24 22:15:52.652049 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:52.652021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8c4mn_e8cb2a34-e342-4afa-8915-ac6efad06772/serve-healthcheck-canary/0.log" Jul 24 22:15:52.745730 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:52.745677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:52.745730 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:52.745728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:52.745950 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:52.745809 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:52.745950 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:52.745824 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:52.745950 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:52.745876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:56.745862618 +0000 UTC m=+79.412873418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:52.745950 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:52.745889 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:15:56.745883123 +0000 UTC m=+79.412893923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : configmap "kube-rbac-proxy" not found Jul 24 22:15:54.743236 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.743201 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85kl8"] Jul 24 22:15:54.745297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.745281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.747320 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.747293 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 22:15:54.747320 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.747310 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 22:15:54.747500 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.747405 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6lqmw\"" Jul 24 22:15:54.747899 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.747882 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 22:15:54.747984 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.747886 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 22:15:54.758301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.758275 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85kl8"] Jul 24 22:15:54.758468 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.758447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwvv\" (UniqueName: \"kubernetes.io/projected/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-kube-api-access-5dwvv\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.758547 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.758533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-key\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.758613 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.758581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-cabundle\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.859657 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.859615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwvv\" (UniqueName: \"kubernetes.io/projected/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-kube-api-access-5dwvv\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.859862 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.859722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-key\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.859862 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.859748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-cabundle\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.860366 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.860339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-cabundle\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.862264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.862246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-signing-key\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:54.868043 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:54.868022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwvv\" (UniqueName: \"kubernetes.io/projected/26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3-kube-api-access-5dwvv\") pod \"service-ca-658d7b7c78-85kl8\" (UID: \"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3\") " pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:55.053802 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:55.053705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" Jul 24 22:15:55.183016 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:55.182982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-85kl8"] Jul 24 22:15:55.186605 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:15:55.186578 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d5f72a_7eb8_4e60_a3eb_fad28f6a16f3.slice/crio-2433aba88604372a32856d5e0520567403820b4d0ef379b741cb342ef552e58b WatchSource:0}: Error finding container 2433aba88604372a32856d5e0520567403820b4d0ef379b741cb342ef552e58b: Status 404 returned error can't find the container with id 2433aba88604372a32856d5e0520567403820b4d0ef379b741cb342ef552e58b Jul 24 22:15:55.258695 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:55.258662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" event={"ID":"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3","Type":"ContainerStarted","Data":"6908277fe4b86e67617a8281f4dabbd059a69c9fbb801be4caa3f735b5e8d957"} Jul 24 22:15:55.258848 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:55.258721 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" event={"ID":"26d5f72a-7eb8-4e60-a3eb-fad28f6a16f3","Type":"ContainerStarted","Data":"2433aba88604372a32856d5e0520567403820b4d0ef379b741cb342ef552e58b"} Jul 24 22:15:55.275000 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:55.274942 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-85kl8" podStartSLOduration=1.274921397 podStartE2EDuration="1.274921397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:55.274645369 +0000 UTC m=+77.941656189" watchObservedRunningTime="2026-07-24 22:15:55.274921397 +0000 UTC m=+77.941932220" Jul 24 22:15:56.774039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:56.773983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:56.774039 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:15:56.774041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:15:56.774520 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:56.774143 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:15:56.774520 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:56.774160 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:15:56.774520 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:56.774227 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:04.774208135 +0000 UTC m=+87.441218942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:15:56.774520 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:15:56.774244 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:04.774235266 +0000 UTC m=+87.441246068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : configmap "kube-rbac-proxy" not found Jul 24 22:16:04.839062 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:04.838919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:04.839062 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:04.838958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:04.839725 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:04.839081 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:16:04.839725 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:04.839151 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls podName:b983e892-a4d5-4ed3-9e28-4bce97b48431 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:20.839133166 +0000 UTC m=+103.506143968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzzjg" (UID: "b983e892-a4d5-4ed3-9e28-4bce97b48431") : secret "insights-runtime-extractor-tls" not found Jul 24 22:16:04.839725 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:04.839620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b983e892-a4d5-4ed3-9e28-4bce97b48431-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:16.318440 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.318410 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b"] Jul 24 22:16:16.322724 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.322704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.325372 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.325353 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 22:16:16.325497 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.325353 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 22:16:16.325497 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.325385 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 22:16:16.326195 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.326181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 22:16:16.342771 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.342742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b"] Jul 24 22:16:16.414731 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.414707 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6"] Jul 24 22:16:16.417400 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.417386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.420390 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.420371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 22:16:16.420476 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.420444 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 22:16:16.420661 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.420646 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fnvwc\"" Jul 24 22:16:16.428220 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.428198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4m9\" (UniqueName: \"kubernetes.io/projected/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-kube-api-access-nc4m9\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.428351 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.428332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-tmp\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.428421 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.428363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-klusterlet-config\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.429280 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.429259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6"] Jul 24 22:16:16.529584 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-tmp\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.529584 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-klusterlet-config\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.529836 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4m9\" (UniqueName: \"kubernetes.io/projected/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-kube-api-access-nc4m9\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.529836 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79b7ae50-984d-4554-b074-afa345084313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.529939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79b7ae50-984d-4554-b074-afa345084313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.529939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.529904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-tmp\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.532160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.532142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-klusterlet-config\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.541599 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.541571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4m9\" (UniqueName: \"kubernetes.io/projected/3c0b763e-7037-4f8c-96ac-966ac09bfcbb-kube-api-access-nc4m9\") pod \"klusterlet-addon-workmgr-788c59ccb-fbz7b\" (UID: \"3c0b763e-7037-4f8c-96ac-966ac09bfcbb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.630891 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.630861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79b7ae50-984d-4554-b074-afa345084313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.631018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.630908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79b7ae50-984d-4554-b074-afa345084313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.631129 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.631106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:16.631516 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.631499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79b7ae50-984d-4554-b074-afa345084313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.633320 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.633301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/79b7ae50-984d-4554-b074-afa345084313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jpnv6\" (UID: \"79b7ae50-984d-4554-b074-afa345084313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.725653 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.725605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" Jul 24 22:16:16.761840 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.761812 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b"] Jul 24 22:16:16.765620 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:16.765594 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0b763e_7037_4f8c_96ac_966ac09bfcbb.slice/crio-80cbe86ea61ebabc976530adda4c42c30e3aa0b78fdf891c5be3bd99d96a505d WatchSource:0}: Error finding container 80cbe86ea61ebabc976530adda4c42c30e3aa0b78fdf891c5be3bd99d96a505d: Status 404 returned error can't find the container with id 80cbe86ea61ebabc976530adda4c42c30e3aa0b78fdf891c5be3bd99d96a505d Jul 24 22:16:16.848503 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.848467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6"] Jul 24 22:16:16.851370 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:16.851343 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b7ae50_984d_4554_b074_afa345084313.slice/crio-60d7e0452322b016f11256a87ee0800346d7988254e14c398a306e34e74cbbda WatchSource:0}: Error finding container 60d7e0452322b016f11256a87ee0800346d7988254e14c398a306e34e74cbbda: Status 404 returned error can't find the container with id 60d7e0452322b016f11256a87ee0800346d7988254e14c398a306e34e74cbbda Jul 24 22:16:16.934452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.934354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:16:16.934452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.934408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:16:16.936821 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.936791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08c4a51f-148d-4a9f-a9cd-dee108d95b40-metrics-tls\") pod \"dns-default-gmhgc\" (UID: \"08c4a51f-148d-4a9f-a9cd-dee108d95b40\") " pod="openshift-dns/dns-default-gmhgc" Jul 24 22:16:16.936910 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.936846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"image-registry-56787b77dc-8fbkn\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:16:16.958786 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.958757 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sr77c\"" Jul 24 22:16:16.967370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.967350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:16:16.984340 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.984320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wnkgk\"" Jul 24 22:16:16.993181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:16.993156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gmhgc" Jul 24 22:16:17.105530 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.105368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:16:17.108741 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:17.108712 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39537f_3022_4277_aabd_60bff510e2bb.slice/crio-a5e0a38712e858fbb2821a9a4a1a4c29e1e0c8800937fd3c3ca5acfacb2be0f6 WatchSource:0}: Error finding container a5e0a38712e858fbb2821a9a4a1a4c29e1e0c8800937fd3c3ca5acfacb2be0f6: Status 404 returned error can't find the container with id a5e0a38712e858fbb2821a9a4a1a4c29e1e0c8800937fd3c3ca5acfacb2be0f6 Jul 24 22:16:17.127191 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.127169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gmhgc"] Jul 24 22:16:17.136484 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:17.136453 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c4a51f_148d_4a9f_a9cd_dee108d95b40.slice/crio-cc535f00d70223535d38e5f334e33d786b9b53edc0e57de85f0abd4ef3ae64e9 WatchSource:0}: Error finding container cc535f00d70223535d38e5f334e33d786b9b53edc0e57de85f0abd4ef3ae64e9: Status 404 returned error can't find the container with id cc535f00d70223535d38e5f334e33d786b9b53edc0e57de85f0abd4ef3ae64e9 Jul 24 22:16:17.326602 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.326504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gmhgc" event={"ID":"08c4a51f-148d-4a9f-a9cd-dee108d95b40","Type":"ContainerStarted","Data":"cc535f00d70223535d38e5f334e33d786b9b53edc0e57de85f0abd4ef3ae64e9"} Jul 24 22:16:17.327704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.327673 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" event={"ID":"3c0b763e-7037-4f8c-96ac-966ac09bfcbb","Type":"ContainerStarted","Data":"80cbe86ea61ebabc976530adda4c42c30e3aa0b78fdf891c5be3bd99d96a505d"} Jul 24 22:16:17.329185 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.329164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" event={"ID":"fc39537f-3022-4277-aabd-60bff510e2bb","Type":"ContainerStarted","Data":"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa"} Jul 24 22:16:17.329279 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.329200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" event={"ID":"fc39537f-3022-4277-aabd-60bff510e2bb","Type":"ContainerStarted","Data":"a5e0a38712e858fbb2821a9a4a1a4c29e1e0c8800937fd3c3ca5acfacb2be0f6"} Jul 24 22:16:17.329279 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.329233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:16:17.330330 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.330311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" event={"ID":"79b7ae50-984d-4554-b074-afa345084313","Type":"ContainerStarted","Data":"60d7e0452322b016f11256a87ee0800346d7988254e14c398a306e34e74cbbda"} Jul 24 22:16:17.364676 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:17.364629 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" podStartSLOduration=99.364613816 podStartE2EDuration="1m39.364613816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:17.363592799 +0000 UTC m=+100.030603621" watchObservedRunningTime="2026-07-24 22:16:17.364613816 +0000 UTC m=+100.031624636" Jul 24 22:16:20.240403 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:20.240368 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vcvw9" Jul 24 22:16:20.871932 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:20.871895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:20.874823 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:20.874792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b983e892-a4d5-4ed3-9e28-4bce97b48431-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzzjg\" (UID: \"b983e892-a4d5-4ed3-9e28-4bce97b48431\") " pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:20.884293 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:20.884273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzzjg" Jul 24 22:16:21.576473 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.576427 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6"] Jul 24 22:16:21.589318 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.589296 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6"] Jul 24 22:16:21.589495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.589383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:21.591743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.591723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 22:16:21.591743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.591738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-npj68\"" Jul 24 22:16:21.682233 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.682197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f412ff1-111f-4042-a67d-7033337d2fb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5jkw6\" (UID: \"7f412ff1-111f-4042-a67d-7033337d2fb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:21.782938 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.782916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f412ff1-111f-4042-a67d-7033337d2fb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5jkw6\" (UID: \"7f412ff1-111f-4042-a67d-7033337d2fb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:21.785297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.785273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f412ff1-111f-4042-a67d-7033337d2fb4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5jkw6\" (UID: \"7f412ff1-111f-4042-a67d-7033337d2fb4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:21.899375 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.899350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:21.933026 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:21.932926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzzjg"] Jul 24 22:16:21.937102 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:21.937070 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb983e892_a4d5_4ed3_9e28_4bce97b48431.slice/crio-05db9bb7587320bd2dc0112e5255fd09e3ca00cf631066e3ac93b15a124121cd WatchSource:0}: Error finding container 05db9bb7587320bd2dc0112e5255fd09e3ca00cf631066e3ac93b15a124121cd: Status 404 returned error can't find the container with id 05db9bb7587320bd2dc0112e5255fd09e3ca00cf631066e3ac93b15a124121cd Jul 24 22:16:22.096045 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.096012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6"] Jul 24 22:16:22.100134 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:22.100111 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f412ff1_111f_4042_a67d_7033337d2fb4.slice/crio-02cc5707ed03b8e1f6d078329f0e47cbd83077fe69e2999301b9e86c0d909559 WatchSource:0}: Error finding container 02cc5707ed03b8e1f6d078329f0e47cbd83077fe69e2999301b9e86c0d909559: Status 404 returned error can't find the container with id 02cc5707ed03b8e1f6d078329f0e47cbd83077fe69e2999301b9e86c0d909559 Jul 24 22:16:22.353021 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.352985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" event={"ID":"3c0b763e-7037-4f8c-96ac-966ac09bfcbb","Type":"ContainerStarted","Data":"cb5e8eb70fabe39e3c2f94e32e79e7d6b2def1b6a0963c70a5c46bf9e8e63266"} Jul 24 22:16:22.353270 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.353169 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:22.354462 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.354431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzzjg" event={"ID":"b983e892-a4d5-4ed3-9e28-4bce97b48431","Type":"ContainerStarted","Data":"05db9bb7587320bd2dc0112e5255fd09e3ca00cf631066e3ac93b15a124121cd"} Jul 24 22:16:22.355068 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.355049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" Jul 24 22:16:22.356487 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.356462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" event={"ID":"79b7ae50-984d-4554-b074-afa345084313","Type":"ContainerStarted","Data":"e5439aa367cf0929f5f15a93a130c4c66d11e242eb479a89ca44966722ee1753"} Jul 24 22:16:22.358666 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.358642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gmhgc" event={"ID":"08c4a51f-148d-4a9f-a9cd-dee108d95b40","Type":"ContainerStarted","Data":"7436ed09e20dcf277d268a238c5159b60165fb8fb87931ddc43a2e24ab5b8739"} Jul 24 22:16:22.358773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.358721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gmhgc" Jul 24 22:16:22.358773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.358750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gmhgc" event={"ID":"08c4a51f-148d-4a9f-a9cd-dee108d95b40","Type":"ContainerStarted","Data":"0b711576656cbf13c31110ac1976d13b4fd224988c2af1d0014daa37ae18b9d0"} Jul 24 22:16:22.359877 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.359858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" event={"ID":"7f412ff1-111f-4042-a67d-7033337d2fb4","Type":"ContainerStarted","Data":"02cc5707ed03b8e1f6d078329f0e47cbd83077fe69e2999301b9e86c0d909559"} Jul 24 22:16:22.369087 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.369049 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-788c59ccb-fbz7b" podStartSLOduration=1.327178631 podStartE2EDuration="6.369035226s" totalImagesPullingTime="5.041856595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:22.368174409 +0000 UTC m=+105.035185232" watchObservedRunningTime="2026-07-24 22:16:22.369035226 +0000 UTC m=+105.036046048" Jul 24 22:16:22.382258 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.382204 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jpnv6" podStartSLOduration=1.46028186 podStartE2EDuration="6.382188523s" totalImagesPullingTime="4.921906663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:22.381799383 +0000 UTC m=+105.048810206" watchObservedRunningTime="2026-07-24 22:16:22.382188523 +0000 UTC m=+105.049199347" Jul 24 22:16:22.411588 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:22.411276 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gmhgc" podStartSLOduration=64.758022383 podStartE2EDuration="1m9.411254318s" totalImagesPullingTime="4.653231935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:22.411178598 +0000 UTC m=+105.078189420" watchObservedRunningTime="2026-07-24 22:16:22.411254318 +0000 UTC m=+105.078265142" Jul 24 22:16:24.368148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:24.368105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" event={"ID":"7f412ff1-111f-4042-a67d-7033337d2fb4","Type":"ContainerStarted","Data":"af99140100e68c91f0a359cd46559b3ba37e094f8ea3dade9bd0df0759ec5f2e"} Jul 24 22:16:24.368630 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:24.368271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:24.373092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:24.373069 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" Jul 24 22:16:24.387342 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:24.387295 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5jkw6" podStartSLOduration=2.089510531 podStartE2EDuration="3.387277481s" totalImagesPullingTime="1.29776695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:24.386506048 +0000 UTC m=+107.053516870" watchObservedRunningTime="2026-07-24 22:16:24.387277481 +0000 UTC m=+107.054288324" Jul 24 22:16:27.382349 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:27.382266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzzjg" event={"ID":"b983e892-a4d5-4ed3-9e28-4bce97b48431","Type":"ContainerStarted","Data":"bd52f99e240ca943840b46d05f1a77404b84771436852aa09e5bc49daf4ed262"} Jul 24 22:16:30.392858 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.392824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzzjg" event={"ID":"b983e892-a4d5-4ed3-9e28-4bce97b48431","Type":"ContainerStarted","Data":"8105228a54e31fc0a4e459cdd998d1ca60d2ecb7be7e4d7b68ad5e75d928fb49"} Jul 24 22:16:30.786465 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.786383 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-dpwjh"] Jul 24 22:16:30.789489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.789466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:30.791806 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.791783 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-brjqb\"" Jul 24 22:16:30.791949 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.791806 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 22:16:30.792203 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.792181 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 22:16:30.800361 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.800309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dpwjh"] Jul 24 22:16:30.964143 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:30.964114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5pz\" (UniqueName: \"kubernetes.io/projected/dea03dea-67ef-4cc9-93ec-cde8df14463d-kube-api-access-2q5pz\") pod \"downloads-7c97dc4555-dpwjh\" (UID: \"dea03dea-67ef-4cc9-93ec-cde8df14463d\") " pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:31.064601 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:31.064504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5pz\" (UniqueName: \"kubernetes.io/projected/dea03dea-67ef-4cc9-93ec-cde8df14463d-kube-api-access-2q5pz\") pod \"downloads-7c97dc4555-dpwjh\" (UID: \"dea03dea-67ef-4cc9-93ec-cde8df14463d\") " pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:31.073302 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:31.073275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5pz\" (UniqueName: \"kubernetes.io/projected/dea03dea-67ef-4cc9-93ec-cde8df14463d-kube-api-access-2q5pz\") pod \"downloads-7c97dc4555-dpwjh\" (UID: \"dea03dea-67ef-4cc9-93ec-cde8df14463d\") " pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:31.100778 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:31.100717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:31.243990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:31.243959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dpwjh"] Jul 24 22:16:31.247759 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:31.247721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea03dea_67ef_4cc9_93ec_cde8df14463d.slice/crio-cdf8c70da0ab042430ba2f87ce8a0aba0befa3f7e96d67d0e66bbbd7446ee8a4 WatchSource:0}: Error finding container cdf8c70da0ab042430ba2f87ce8a0aba0befa3f7e96d67d0e66bbbd7446ee8a4: Status 404 returned error can't find the container with id cdf8c70da0ab042430ba2f87ce8a0aba0befa3f7e96d67d0e66bbbd7446ee8a4 Jul 24 22:16:31.397979 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:31.397942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dpwjh" event={"ID":"dea03dea-67ef-4cc9-93ec-cde8df14463d","Type":"ContainerStarted","Data":"cdf8c70da0ab042430ba2f87ce8a0aba0befa3f7e96d67d0e66bbbd7446ee8a4"} Jul 24 22:16:32.366199 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:32.365897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gmhgc" Jul 24 22:16:33.024834 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.023449 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg"] Jul 24 22:16:33.036092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.036024 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg"] Jul 24 22:16:33.036092 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.036047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.040018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.039998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 22:16:33.048083 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.048061 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9xzsl\"" Jul 24 22:16:33.048279 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.048264 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 22:16:33.048451 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.048436 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 22:16:33.048658 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.048641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 22:16:33.050094 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.048835 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 22:16:33.053593 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.053053 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rfsvk"] Jul 24 22:16:33.056642 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.056624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.059026 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.059010 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 22:16:33.059466 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.059295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gv6z9\"" Jul 24 22:16:33.059856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.059678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 22:16:33.061848 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.061825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zm7\" (UniqueName: \"kubernetes.io/projected/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-kube-api-access-t6zm7\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-wtmp\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-textfile\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-sys\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-metrics-client-ca\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.183983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsxb\" (UniqueName: \"kubernetes.io/projected/80ccf359-893c-4094-bd87-ca216bb2ec75-kube-api-access-cxsxb\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.184028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.184147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.184056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-root\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-textfile\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-sys\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-metrics-client-ca\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsxb\" (UniqueName: \"kubernetes.io/projected/80ccf359-893c-4094-bd87-ca216bb2ec75-kube-api-access-cxsxb\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.284990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-root\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.285055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zm7\" (UniqueName: \"kubernetes.io/projected/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-kube-api-access-t6zm7\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.285088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.285120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-wtmp\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.285275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-wtmp\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286046 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.285784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-sys\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286948 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.286173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-textfile\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286948 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.286261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/80ccf359-893c-4094-bd87-ca216bb2ec75-root\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.286948 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:33.286645 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 22:16:33.286948 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:33.286710 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls podName:80ccf359-893c-4094-bd87-ca216bb2ec75 nodeName:}" failed. No retries permitted until 2026-07-24 22:16:33.786691035 +0000 UTC m=+116.453701834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls") pod "node-exporter-rfsvk" (UID: "80ccf359-893c-4094-bd87-ca216bb2ec75") : secret "node-exporter-tls" not found Jul 24 22:16:33.287151 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.286995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.287848 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.287633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ccf359-893c-4094-bd87-ca216bb2ec75-metrics-client-ca\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.287848 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.287788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.291077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.291048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.292988 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.292948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.294205 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.294167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.310576 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.302659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zm7\" (UniqueName: \"kubernetes.io/projected/bdd5fb11-6bb2-4c2e-9406-0f22a90bb803-kube-api-access-t6zm7\") pod \"openshift-state-metrics-b8f49f5c6-nzmtg\" (UID: \"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.311461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.311312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsxb\" (UniqueName: \"kubernetes.io/projected/80ccf359-893c-4094-bd87-ca216bb2ec75-kube-api-access-cxsxb\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.366510 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.366278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" Jul 24 22:16:33.789739 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.789700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.795289 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.795236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/80ccf359-893c-4094-bd87-ca216bb2ec75-node-exporter-tls\") pod \"node-exporter-rfsvk\" (UID: \"80ccf359-893c-4094-bd87-ca216bb2ec75\") " pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:33.976233 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:33.976193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfsvk" Jul 24 22:16:34.336637 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:34.336602 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ccf359_893c_4094_bd87_ca216bb2ec75.slice/crio-876c15481601877f86944f772611cf81e89abf5a8fd8108c0301278efc89aedb WatchSource:0}: Error finding container 876c15481601877f86944f772611cf81e89abf5a8fd8108c0301278efc89aedb: Status 404 returned error can't find the container with id 876c15481601877f86944f772611cf81e89abf5a8fd8108c0301278efc89aedb Jul 24 22:16:34.418669 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:34.418633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsvk" event={"ID":"80ccf359-893c-4094-bd87-ca216bb2ec75","Type":"ContainerStarted","Data":"876c15481601877f86944f772611cf81e89abf5a8fd8108c0301278efc89aedb"} Jul 24 22:16:34.481068 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:34.481038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg"] Jul 24 22:16:34.484031 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:34.484001 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd5fb11_6bb2_4c2e_9406_0f22a90bb803.slice/crio-bf299f079bd14bbb600da8d156943a0c5d5e30ee8114662342fcb104947b5886 WatchSource:0}: Error finding container bf299f079bd14bbb600da8d156943a0c5d5e30ee8114662342fcb104947b5886: Status 404 returned error can't find the container with id bf299f079bd14bbb600da8d156943a0c5d5e30ee8114662342fcb104947b5886 Jul 24 22:16:35.002850 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.002799 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b84699cd5-fqqjk"] Jul 24 22:16:35.007463 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.007357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.009924 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.009773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 22:16:35.009924 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.009846 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 22:16:35.010297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.010277 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 22:16:35.010425 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.010361 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 22:16:35.010425 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.010379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b2lmn\"" Jul 24 22:16:35.010849 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.010826 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9tm7n2vdk1oko\"" Jul 24 22:16:35.012824 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.012653 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 22:16:35.029586 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.029532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b84699cd5-fqqjk"] Jul 24 22:16:35.100499 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100499 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-grpc-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdqw\" (UniqueName: \"kubernetes.io/projected/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-kube-api-access-vkdqw\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.100764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.100730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-metrics-client-ca\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.201994 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.201965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-grpc-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202182 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdqw\" (UniqueName: \"kubernetes.io/projected/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-kube-api-access-vkdqw\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202327 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-metrics-client-ca\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202395 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202448 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202448 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.202590 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.202457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.203401 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.203202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-metrics-client-ca\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.205644 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.205597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.205748 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.205720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-grpc-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.206215 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.206180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.206783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.206743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-tls\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.206957 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.206933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.208048 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.208028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.212051 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.212030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdqw\" (UniqueName: \"kubernetes.io/projected/c0666bc7-fe4a-4e2b-ae81-1ceb823561ee-kube-api-access-vkdqw\") pod \"thanos-querier-b84699cd5-fqqjk\" (UID: \"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee\") " pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.319497 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.319472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:35.433974 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.433322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsvk" event={"ID":"80ccf359-893c-4094-bd87-ca216bb2ec75","Type":"ContainerStarted","Data":"53d20688094c80156e62abe8ff90422ce90b443d5bc4ca458d96711bd4dd7890"} Jul 24 22:16:35.438694 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.438665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" event={"ID":"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803","Type":"ContainerStarted","Data":"146b17be9d42b1d21e7adb89c038d8d4c03f68ddf632328835f9010f9f791873"} Jul 24 22:16:35.438812 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.438728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" event={"ID":"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803","Type":"ContainerStarted","Data":"d77e5b8fbfafc75dd23f1944b2d434353357d93d667fb40eff6ace0cf275f880"} Jul 24 22:16:35.438812 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.438756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" event={"ID":"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803","Type":"ContainerStarted","Data":"bf299f079bd14bbb600da8d156943a0c5d5e30ee8114662342fcb104947b5886"} Jul 24 22:16:35.441412 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.441386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzzjg" event={"ID":"b983e892-a4d5-4ed3-9e28-4bce97b48431","Type":"ContainerStarted","Data":"11548bac3d8957cba20faef594adfff38247c73356705f16d0018528f9b2fbe0"} Jul 24 22:16:35.496652 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.496600 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lzzjg" podStartSLOduration=34.150267834 podStartE2EDuration="46.496578097s" totalImagesPullingTime="12.346310263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:15:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:35.495379079 +0000 UTC m=+118.162389901" watchObservedRunningTime="2026-07-24 22:16:35.496578097 +0000 UTC m=+118.163588919" Jul 24 22:16:35.512855 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:35.512762 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b84699cd5-fqqjk"] Jul 24 22:16:35.516000 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:35.514670 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0666bc7_fe4a_4e2b_ae81_1ceb823561ee.slice/crio-bc2623c3170853156e768e8ad39cb73d6bcab340ac3b8f990f845feaf8f20751 WatchSource:0}: Error finding container bc2623c3170853156e768e8ad39cb73d6bcab340ac3b8f990f845feaf8f20751: Status 404 returned error can't find the container with id bc2623c3170853156e768e8ad39cb73d6bcab340ac3b8f990f845feaf8f20751 Jul 24 22:16:36.448094 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:36.447985 2576 generic.go:356] "Generic (PLEG): container finished" podID="80ccf359-893c-4094-bd87-ca216bb2ec75" containerID="53d20688094c80156e62abe8ff90422ce90b443d5bc4ca458d96711bd4dd7890" exitCode=0 Jul 24 22:16:36.448094 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:36.448042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsvk" event={"ID":"80ccf359-893c-4094-bd87-ca216bb2ec75","Type":"ContainerDied","Data":"53d20688094c80156e62abe8ff90422ce90b443d5bc4ca458d96711bd4dd7890"} Jul 24 22:16:36.451453 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:36.450920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" event={"ID":"bdd5fb11-6bb2-4c2e-9406-0f22a90bb803","Type":"ContainerStarted","Data":"7d8250baab522c054750e5dbf040dbba0507249e1965cc152f03cb54ef39ed98"} Jul 24 22:16:36.453279 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:36.452781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"bc2623c3170853156e768e8ad39cb73d6bcab340ac3b8f990f845feaf8f20751"} Jul 24 22:16:36.499691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:36.499271 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-nzmtg" podStartSLOduration=3.197567936 podStartE2EDuration="4.499249305s" totalImagesPullingTime="1.301681369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:36.497906396 +0000 UTC m=+119.164917212" watchObservedRunningTime="2026-07-24 22:16:36.499249305 +0000 UTC m=+119.166260128" Jul 24 22:16:37.461248 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.461212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsvk" event={"ID":"80ccf359-893c-4094-bd87-ca216bb2ec75","Type":"ContainerStarted","Data":"988cfcbe3aaddbd630c65688473bd3c431f8322f3cc58c2ed21a0bcbde7692e7"} Jul 24 22:16:37.461754 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.461303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfsvk" event={"ID":"80ccf359-893c-4094-bd87-ca216bb2ec75","Type":"ContainerStarted","Data":"f56f924645c82d4d5dddc2f2cb04d5f470519aba6e375bfb2822a81155ec0a0d"} Jul 24 22:16:37.492874 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.492800 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rfsvk" podStartSLOduration=3.452255796 podStartE2EDuration="4.492777985s" totalImagesPullingTime="938.7202ms" totalInitContainerRuntime="101.801989ms" isStatefulPod=true podCreationTimestamp="2026-07-24 22:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:37.490730641 +0000 UTC m=+120.157741463" watchObservedRunningTime="2026-07-24 22:16:37.492777985 +0000 UTC m=+120.159788804" Jul 24 22:16:37.507808 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.507772 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78978c6859-fb6rg"] Jul 24 22:16:37.511060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.510981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.514230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514086 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 22:16:37.514230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 22:16:37.514230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 22:16:37.514471 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 22:16:37.514471 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514336 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fgrtlv0e84v8v\"" Jul 24 22:16:37.514471 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.514380 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wrz2\"" Jul 24 22:16:37.527031 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.526972 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78978c6859-fb6rg"] Jul 24 22:16:37.623892 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.623857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-client-certs\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624033 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.623899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-client-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624033 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.624005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-metrics-server-audit-profiles\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624143 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.624081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-audit-log\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624143 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.624124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwn47\" (UniqueName: \"kubernetes.io/projected/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-kube-api-access-qwn47\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624243 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.624219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-tls\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.624311 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.624287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwn47\" (UniqueName: \"kubernetes.io/projected/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-kube-api-access-qwn47\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-tls\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-client-certs\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-client-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.725989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-metrics-server-audit-profiles\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.726026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-audit-log\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.727077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.726399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-audit-log\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.728297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.728267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.728697 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.728670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-metrics-server-audit-profiles\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.731347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.730983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-client-certs\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.732357 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.732332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-secret-metrics-server-tls\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.732675 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.732632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-client-ca-bundle\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.737679 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.737629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwn47\" (UniqueName: \"kubernetes.io/projected/6b7dec87-0f5b-472c-8aea-1d7b02338b7c-kube-api-access-qwn47\") pod \"metrics-server-78978c6859-fb6rg\" (UID: \"6b7dec87-0f5b-472c-8aea-1d7b02338b7c\") " pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.750234 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.750207 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv"] Jul 24 22:16:37.754834 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.754760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:37.757825 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.756935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 22:16:37.757825 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.756978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bdjkm\"" Jul 24 22:16:37.765344 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.765311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv"] Jul 24 22:16:37.824952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.824909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wrz2\"" Jul 24 22:16:37.833203 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.833166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:37.928203 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.928166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2bt\" (UniqueName: \"kubernetes.io/projected/0dfae74e-ccb6-4393-8ef0-20459106cb1a-kube-api-access-6z2bt\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:37.928363 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:37.928236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.004583 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.004483 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78978c6859-fb6rg"] Jul 24 22:16:38.008212 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:38.008174 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7dec87_0f5b_472c_8aea_1d7b02338b7c.slice/crio-cb5e39c4a2ee4115da7f738f113f931caa02816907b763827b82e4ca4f45ee39 WatchSource:0}: Error finding container cb5e39c4a2ee4115da7f738f113f931caa02816907b763827b82e4ca4f45ee39: Status 404 returned error can't find the container with id cb5e39c4a2ee4115da7f738f113f931caa02816907b763827b82e4ca4f45ee39 Jul 24 22:16:38.029360 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.029329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2bt\" (UniqueName: \"kubernetes.io/projected/0dfae74e-ccb6-4393-8ef0-20459106cb1a-kube-api-access-6z2bt\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.029484 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.029393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.029606 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:38.029584 2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 22:16:38.029677 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:16:38.029661 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert podName:0dfae74e-ccb6-4393-8ef0-20459106cb1a nodeName:}" failed. No retries permitted until 2026-07-24 22:16:38.529637976 +0000 UTC m=+121.196648776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-9zbvv" (UID: "0dfae74e-ccb6-4393-8ef0-20459106cb1a") : secret "monitoring-plugin-cert" not found Jul 24 22:16:38.038598 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.038548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2bt\" (UniqueName: \"kubernetes.io/projected/0dfae74e-ccb6-4393-8ef0-20459106cb1a-kube-api-access-6z2bt\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.342147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.342077 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:16:38.468538 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.468490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"e10949417daa56b9d9d43cc3d1150b7ea3f1d28dcb2357a2d397683d90bc672a"} Jul 24 22:16:38.469007 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.468614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"f8862d64e5ab9e0d500ba793b35e83faf8d31a19520003e606bde4161ef9e878"} Jul 24 22:16:38.469007 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.468656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"15119096bfc43eb5faa097866901a4572989ed1a47cda6d0304d1eb5cd2374ef"} Jul 24 22:16:38.469881 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.469797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" event={"ID":"6b7dec87-0f5b-472c-8aea-1d7b02338b7c","Type":"ContainerStarted","Data":"cb5e39c4a2ee4115da7f738f113f931caa02816907b763827b82e4ca4f45ee39"} Jul 24 22:16:38.534716 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.533782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.537202 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.537169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0dfae74e-ccb6-4393-8ef0-20459106cb1a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-9zbvv\" (UID: \"0dfae74e-ccb6-4393-8ef0-20459106cb1a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.571891 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.571854 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:16:38.679939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.679907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:38.833143 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:38.833102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv"] Jul 24 22:16:38.836345 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:38.836313 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfae74e_ccb6_4393_8ef0_20459106cb1a.slice/crio-3d95d739dae9510d94c0cac66212a4dc72bdd89557c5d2804175a0a530130211 WatchSource:0}: Error finding container 3d95d739dae9510d94c0cac66212a4dc72bdd89557c5d2804175a0a530130211: Status 404 returned error can't find the container with id 3d95d739dae9510d94c0cac66212a4dc72bdd89557c5d2804175a0a530130211 Jul 24 22:16:39.295166 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.295130 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:16:39.299520 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.299495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.301707 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.301679 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 22:16:39.301848 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.301743 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 22:16:39.301991 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.301973 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 22:16:39.302070 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.301971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 22:16:39.302532 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.302512 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n96r2\"" Jul 24 22:16:39.302996 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.302971 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4cfl49k0v7hsq\"" Jul 24 22:16:39.303230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 22:16:39.303230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303190 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 22:16:39.303230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303223 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 22:16:39.303412 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303369 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 22:16:39.303463 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303437 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 22:16:39.303795 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303778 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 22:16:39.304030 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.303949 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 22:16:39.304508 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.304292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 22:16:39.304508 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.304321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 22:16:39.305202 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.305181 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 22:16:39.319769 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.319742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:16:39.341937 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.341907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.341942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.341970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342262 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342262 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342362 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342362 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342440 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jf2\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342510 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342510 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342615 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342615 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342615 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342762 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342762 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342762 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342762 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.342890 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.342787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.443492 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.443445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444038 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jf2\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444346 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.444827 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.444799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.445932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.446823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.447477 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.447061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.448054 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.447546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.448903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.448781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.448903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.448854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.449525 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.449497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.449915 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.449890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.450398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.450334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.452314 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.452103 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.452314 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.452187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.452314 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.452275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.452796 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.452755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.453433 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.453217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.453433 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.453395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.453605 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.453442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.454005 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.453965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.454241 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.454185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.455103 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.455070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.455771 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.455745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jf2\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.456958 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.456904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config\") pod \"prometheus-k8s-0\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:39.479469 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.479429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"d69d2cdd27b50edc86b90339e97fe3ef90429e13b9c73129280ead323c9c14ab"} Jul 24 22:16:39.479958 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.479575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"547ba4855c9a9acaa910c01d90b42b52691ccb47e90936b4fdf238df6a64459b"} Jul 24 22:16:39.479958 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.479621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" event={"ID":"c0666bc7-fe4a-4e2b-ae81-1ceb823561ee","Type":"ContainerStarted","Data":"b00be80e9686a637701a4209f8d7e58a10eb5557c72c4eb590bc4cb31b84e4b7"} Jul 24 22:16:39.480250 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.480231 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:39.482372 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.482343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" event={"ID":"0dfae74e-ccb6-4393-8ef0-20459106cb1a","Type":"ContainerStarted","Data":"3d95d739dae9510d94c0cac66212a4dc72bdd89557c5d2804175a0a530130211"} Jul 24 22:16:39.504682 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.503317 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" podStartSLOduration=2.586376197 podStartE2EDuration="5.503292953s" totalImagesPullingTime="2.916916756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:39.500689452 +0000 UTC m=+122.167700275" watchObservedRunningTime="2026-07-24 22:16:39.503292953 +0000 UTC m=+122.170303775" Jul 24 22:16:39.613070 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:39.612920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:40.489443 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.489369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" event={"ID":"0dfae74e-ccb6-4393-8ef0-20459106cb1a","Type":"ContainerStarted","Data":"23af7572eb01a736eee84e8d7ca74297ff4afe19e6d7ca079f2d5cb947472e1b"} Jul 24 22:16:40.490108 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.490078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:40.492045 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.491990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" event={"ID":"6b7dec87-0f5b-472c-8aea-1d7b02338b7c","Type":"ContainerStarted","Data":"1029e19eaa8f5175ae8c8feacb94d97689d09dc03c1ee4ae7cd03f6299f911d1"} Jul 24 22:16:40.497148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.497122 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" Jul 24 22:16:40.504446 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.504418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:16:40.506903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.506852 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-9zbvv" podStartSLOduration=2.010920244 podStartE2EDuration="3.506832808s" totalImagesPullingTime="1.495912564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:40.506016604 +0000 UTC m=+123.173027428" watchObservedRunningTime="2026-07-24 22:16:40.506832808 +0000 UTC m=+123.173843631" Jul 24 22:16:40.510103 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:40.510065 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efd543c_eb65_4464_95d5_374b6baf01dc.slice/crio-888d552330fc0b457043dd613107c37a310a862007e5503a8316d7274eedb4cf WatchSource:0}: Error finding container 888d552330fc0b457043dd613107c37a310a862007e5503a8316d7274eedb4cf: Status 404 returned error can't find the container with id 888d552330fc0b457043dd613107c37a310a862007e5503a8316d7274eedb4cf Jul 24 22:16:40.522535 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:40.522485 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" podStartSLOduration=1.200906791 podStartE2EDuration="3.522467066s" totalImagesPullingTime="2.321560275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:40.52111622 +0000 UTC m=+123.188127044" watchObservedRunningTime="2026-07-24 22:16:40.522467066 +0000 UTC m=+123.189477887" Jul 24 22:16:41.501619 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:41.501438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"888d552330fc0b457043dd613107c37a310a862007e5503a8316d7274eedb4cf"} Jul 24 22:16:42.508107 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:42.508062 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="55285098b83350d04c11e43090eae78d5397465a9ba9ed5fdd84993135b4e9d2" exitCode=0 Jul 24 22:16:42.508579 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:42.508162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"55285098b83350d04c11e43090eae78d5397465a9ba9ed5fdd84993135b4e9d2"} Jul 24 22:16:45.499122 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:45.499091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b84699cd5-fqqjk" Jul 24 22:16:47.634160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:47.634118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:16:47.636763 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:47.636737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1654044-8118-4cc6-8743-397a4adab49f-metrics-certs\") pod \"network-metrics-daemon-7vhmh\" (UID: \"f1654044-8118-4cc6-8743-397a4adab49f\") " pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:16:47.913472 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:47.913382 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-767d7\"" Jul 24 22:16:47.921875 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:47.921842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vhmh" Jul 24 22:16:49.292405 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:49.292378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vhmh"] Jul 24 22:16:49.444074 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:16:49.443993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1654044_8118_4cc6_8743_397a4adab49f.slice/crio-72c567f822a6d7c1e0863e873df66f575e00a76502b3fa3df39e92b4aa23b733 WatchSource:0}: Error finding container 72c567f822a6d7c1e0863e873df66f575e00a76502b3fa3df39e92b4aa23b733: Status 404 returned error can't find the container with id 72c567f822a6d7c1e0863e873df66f575e00a76502b3fa3df39e92b4aa23b733 Jul 24 22:16:49.546871 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:49.546835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhmh" event={"ID":"f1654044-8118-4cc6-8743-397a4adab49f","Type":"ContainerStarted","Data":"72c567f822a6d7c1e0863e873df66f575e00a76502b3fa3df39e92b4aa23b733"} Jul 24 22:16:50.556693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:50.556649 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dpwjh" event={"ID":"dea03dea-67ef-4cc9-93ec-cde8df14463d","Type":"ContainerStarted","Data":"3ed6feae07f5ebbd108cd2b284df083bf720e3ea80cec643991a03b6d8f31cf0"} Jul 24 22:16:50.557485 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:50.557457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:50.567605 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:50.567547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-dpwjh" Jul 24 22:16:50.596640 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:50.595865 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-dpwjh" podStartSLOduration=2.318156365 podStartE2EDuration="20.595841452s" totalImagesPullingTime="18.277685087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:50.574735593 +0000 UTC m=+133.241746416" watchObservedRunningTime="2026-07-24 22:16:50.595841452 +0000 UTC m=+133.262852276" Jul 24 22:16:53.574220 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:53.574187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhmh" event={"ID":"f1654044-8118-4cc6-8743-397a4adab49f","Type":"ContainerStarted","Data":"218c0e3ec119e81f388d3ca7a88e4d8216d9a2a206fe766a1fa761b9ed87515c"} Jul 24 22:16:53.574728 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:53.574238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vhmh" event={"ID":"f1654044-8118-4cc6-8743-397a4adab49f","Type":"ContainerStarted","Data":"6cc0980b4f0efe305512f8fe228b486b7c2f4bbac1142b1f4fa7a9e6a3b03c7e"} Jul 24 22:16:53.578088 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:53.578052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"a5f88e3f2f2f2693eb176736c113c3d2e65ed5c6ddd2a5b16ff0826393b49b69"} Jul 24 22:16:53.578162 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:53.578132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"d3f5454f41066a17950bb915b08f3b4c8dc8a08b4c768c26908614c5ebedf9d6"} Jul 24 22:16:53.591573 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:53.591494 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vhmh" podStartSLOduration=132.350010385 podStartE2EDuration="2m15.591469643s" totalImagesPullingTime="3.241459258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:53.589673233 +0000 UTC m=+136.256684056" watchObservedRunningTime="2026-07-24 22:16:53.591469643 +0000 UTC m=+136.258480466" Jul 24 22:16:54.587406 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.587366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"8d27b159a9bf86ba32b7c4947a6beef61db21d08ba82fbf408d3c6e16636122e"} Jul 24 22:16:54.587907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.587446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"d49c56c493986f1ff50e4b05a33910982dc1e63cac795de792fb36e75a731660"} Jul 24 22:16:54.587907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.587492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"5ca4f3d11978810b169282bc0cdbb26b7ddf74b038c4f73af40d8d9545bdc659"} Jul 24 22:16:54.587907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.587532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerStarted","Data":"5a480f69ff185e0daa48be3065e8fbbee49fa3dda9b7d4ddd861bc1379dc8d65"} Jul 24 22:16:54.613245 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.613208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:16:54.613852 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:54.613801 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.572907534 podStartE2EDuration="15.613768617s" totalImagesPullingTime="12.016345226s" totalInitContainerRuntime="24.515857ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:16:54.613148269 +0000 UTC m=+137.280159128" watchObservedRunningTime="2026-07-24 22:16:54.613768617 +0000 UTC m=+137.280779441" Jul 24 22:16:57.834347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:57.834306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:16:57.834981 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:16:57.834395 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:17:03.600706 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.600633 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" podUID="fc39537f-3022-4277-aabd-60bff510e2bb" containerName="registry" containerID="cri-o://a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa" gracePeriod=30 Jul 24 22:17:03.880592 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.880486 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:17:03.996375 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996331 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9vkv\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996375 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996646 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996646 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996646 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996705 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996738 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996790 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration\") pod \"fc39537f-3022-4277-aabd-60bff510e2bb\" (UID: \"fc39537f-3022-4277-aabd-60bff510e2bb\") " Jul 24 22:17:03.996964 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.996883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:03.997048 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.997024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:03.997183 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.997104 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-trusted-ca\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:03.997309 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.997270 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc39537f-3022-4277-aabd-60bff510e2bb-registry-certificates\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:03.999122 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.999064 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv" (OuterVolumeSpecName: "kube-api-access-n9vkv") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "kube-api-access-n9vkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:03.999223 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.999118 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:03.999307 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.999282 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:03.999462 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.999442 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:03.999462 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:03.999446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:04.005527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.005502 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fc39537f-3022-4277-aabd-60bff510e2bb" (UID: "fc39537f-3022-4277-aabd-60bff510e2bb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:17:04.098024 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.097981 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vkv\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-kube-api-access-n9vkv\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.098024 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.098015 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-installation-pull-secrets\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.098024 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.098032 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-bound-sa-token\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.098267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.098045 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc39537f-3022-4277-aabd-60bff510e2bb-ca-trust-extracted\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.098267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.098056 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc39537f-3022-4277-aabd-60bff510e2bb-registry-tls\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.098267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.098069 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc39537f-3022-4277-aabd-60bff510e2bb-image-registry-private-configuration\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:04.631911 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.631870 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc39537f-3022-4277-aabd-60bff510e2bb" containerID="a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa" exitCode=0 Jul 24 22:17:04.632414 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.631926 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" Jul 24 22:17:04.632414 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.631960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" event={"ID":"fc39537f-3022-4277-aabd-60bff510e2bb","Type":"ContainerDied","Data":"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa"} Jul 24 22:17:04.632414 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.632017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56787b77dc-8fbkn" event={"ID":"fc39537f-3022-4277-aabd-60bff510e2bb","Type":"ContainerDied","Data":"a5e0a38712e858fbb2821a9a4a1a4c29e1e0c8800937fd3c3ca5acfacb2be0f6"} Jul 24 22:17:04.632414 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.632054 2576 scope.go:117] "RemoveContainer" containerID="a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa" Jul 24 22:17:04.641628 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.641607 2576 scope.go:117] "RemoveContainer" containerID="a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa" Jul 24 22:17:04.641956 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:17:04.641934 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa\": container with ID starting with a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa not found: ID does not exist" containerID="a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa" Jul 24 22:17:04.642024 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.641969 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa"} err="failed to get container status \"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa\": rpc error: code = NotFound desc = could not find container \"a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa\": container with ID starting with a5e6a133ff12ec8deb920480fa2f07afee4970dfb7847b97dfd07fe97361d9aa not found: ID does not exist" Jul 24 22:17:04.653184 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.653151 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:17:04.659155 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:04.659129 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56787b77dc-8fbkn"] Jul 24 22:17:05.889697 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:05.889666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc39537f-3022-4277-aabd-60bff510e2bb" path="/var/lib/kubelet/pods/fc39537f-3022-4277-aabd-60bff510e2bb/volumes" Jul 24 22:17:17.839522 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:17.839489 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:17:17.843742 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:17.843712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78978c6859-fb6rg" Jul 24 22:17:26.723938 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:26.723902 2576 generic.go:356] "Generic (PLEG): container finished" podID="e448e3b7-5558-42e8-bb21-78cb00b9898c" containerID="854683974a5d884f5f1bff0272de6e6f509138a5002da071f9f0fe74e9852289" exitCode=0 Jul 24 22:17:26.724348 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:26.723979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" event={"ID":"e448e3b7-5558-42e8-bb21-78cb00b9898c","Type":"ContainerDied","Data":"854683974a5d884f5f1bff0272de6e6f509138a5002da071f9f0fe74e9852289"} Jul 24 22:17:26.724348 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:26.724286 2576 scope.go:117] "RemoveContainer" containerID="854683974a5d884f5f1bff0272de6e6f509138a5002da071f9f0fe74e9852289" Jul 24 22:17:27.730136 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:27.730100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bl59f" event={"ID":"e448e3b7-5558-42e8-bb21-78cb00b9898c","Type":"ContainerStarted","Data":"924b798e1e463e00cab36d62c7fcaeb70aaa20ad752d01c77b5ca03539642eea"} Jul 24 22:17:39.614236 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:39.614196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:39.680712 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:39.680684 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:39.791408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:39.791386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:57.609379 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609347 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609688 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="prometheus" containerID="cri-o://d3f5454f41066a17950bb915b08f3b4c8dc8a08b4c768c26908614c5ebedf9d6" gracePeriod=600 Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609718 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy" containerID="cri-o://d49c56c493986f1ff50e4b05a33910982dc1e63cac795de792fb36e75a731660" gracePeriod=600 Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609737 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="config-reloader" containerID="cri-o://a5f88e3f2f2f2693eb176736c113c3d2e65ed5c6ddd2a5b16ff0826393b49b69" gracePeriod=600 Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609748 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-web" containerID="cri-o://5ca4f3d11978810b169282bc0cdbb26b7ddf74b038c4f73af40d8d9545bdc659" gracePeriod=600 Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609807 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8d27b159a9bf86ba32b7c4947a6beef61db21d08ba82fbf408d3c6e16636122e" gracePeriod=600 Jul 24 22:17:57.609947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.609721 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="thanos-sidecar" containerID="cri-o://5a480f69ff185e0daa48be3065e8fbbee49fa3dda9b7d4ddd861bc1379dc8d65" gracePeriod=600 Jul 24 22:17:57.852452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852420 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="8d27b159a9bf86ba32b7c4947a6beef61db21d08ba82fbf408d3c6e16636122e" exitCode=0 Jul 24 22:17:57.852452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852448 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="d49c56c493986f1ff50e4b05a33910982dc1e63cac795de792fb36e75a731660" exitCode=0 Jul 24 22:17:57.852452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852454 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="5ca4f3d11978810b169282bc0cdbb26b7ddf74b038c4f73af40d8d9545bdc659" exitCode=0 Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852460 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="5a480f69ff185e0daa48be3065e8fbbee49fa3dda9b7d4ddd861bc1379dc8d65" exitCode=0 Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852465 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="a5f88e3f2f2f2693eb176736c113c3d2e65ed5c6ddd2a5b16ff0826393b49b69" exitCode=0 Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852469 2576 generic.go:356] "Generic (PLEG): container finished" podID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerID="d3f5454f41066a17950bb915b08f3b4c8dc8a08b4c768c26908614c5ebedf9d6" exitCode=0 Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"8d27b159a9bf86ba32b7c4947a6beef61db21d08ba82fbf408d3c6e16636122e"} Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"d49c56c493986f1ff50e4b05a33910982dc1e63cac795de792fb36e75a731660"} Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"5ca4f3d11978810b169282bc0cdbb26b7ddf74b038c4f73af40d8d9545bdc659"} Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"5a480f69ff185e0daa48be3065e8fbbee49fa3dda9b7d4ddd861bc1379dc8d65"} Jul 24 22:17:57.852703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"a5f88e3f2f2f2693eb176736c113c3d2e65ed5c6ddd2a5b16ff0826393b49b69"} Jul 24 22:17:57.853048 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.852726 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"d3f5454f41066a17950bb915b08f3b4c8dc8a08b4c768c26908614c5ebedf9d6"} Jul 24 22:17:57.856710 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.856691 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:57.949332 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949332 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949291 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949332 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949589 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949589 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949475 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949589 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949502 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949589 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949589 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949586 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jf2\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949636 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949682 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949738 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949772 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.949811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949802 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949837 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.949973 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.950004 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0\") pod \"0efd543c-eb65-4464-95d5-374b6baf01dc\" (UID: \"0efd543c-eb65-4464-95d5-374b6baf01dc\") " Jul 24 22:17:57.950403 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.950197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.950403 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.950222 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951681 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951789 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951812 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951827 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:57.951952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.951843 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-metrics-client-ca\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:57.952446 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.952414 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:17:57.952795 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.952643 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.952795 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.952764 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.953091 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.953062 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:17:57.953170 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.953100 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config" (OuterVolumeSpecName: "config") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.953618 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.953492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.953618 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.953589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2" (OuterVolumeSpecName: "kube-api-access-97jf2") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "kube-api-access-97jf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:57.953764 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.953639 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:17:57.954061 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.954027 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.954194 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.954169 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out" (OuterVolumeSpecName: "config-out") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:17:57.954395 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.954368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.954610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.954582 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.955336 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.955313 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.955491 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.955468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.955615 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.955572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:57.964961 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:57.964940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config" (OuterVolumeSpecName: "web-config") pod "0efd543c-eb65-4464-95d5-374b6baf01dc" (UID: "0efd543c-eb65-4464-95d5-374b6baf01dc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:17:58.052474 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052447 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-tls-assets\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052474 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052473 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052487 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052502 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052515 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052527 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-web-config\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052538 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052577 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-config-out\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052593 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-metrics-client-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052605 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-config\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052617 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jf2\" (UniqueName: \"kubernetes.io/projected/0efd543c-eb65-4464-95d5-374b6baf01dc-kube-api-access-97jf2\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052630 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052645 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efd543c-eb65-4464-95d5-374b6baf01dc-prometheus-k8s-db\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052658 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-grpc-tls\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.052681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052674 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.053064 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.052687 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efd543c-eb65-4464-95d5-374b6baf01dc-secret-kube-rbac-proxy\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:17:58.860119 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.860084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efd543c-eb65-4464-95d5-374b6baf01dc","Type":"ContainerDied","Data":"888d552330fc0b457043dd613107c37a310a862007e5503a8316d7274eedb4cf"} Jul 24 22:17:58.860524 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.860121 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:58.860524 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.860163 2576 scope.go:117] "RemoveContainer" containerID="8d27b159a9bf86ba32b7c4947a6beef61db21d08ba82fbf408d3c6e16636122e" Jul 24 22:17:58.868192 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.868172 2576 scope.go:117] "RemoveContainer" containerID="d49c56c493986f1ff50e4b05a33910982dc1e63cac795de792fb36e75a731660" Jul 24 22:17:58.875693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.875656 2576 scope.go:117] "RemoveContainer" containerID="5ca4f3d11978810b169282bc0cdbb26b7ddf74b038c4f73af40d8d9545bdc659" Jul 24 22:17:58.882548 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.882531 2576 scope.go:117] "RemoveContainer" containerID="5a480f69ff185e0daa48be3065e8fbbee49fa3dda9b7d4ddd861bc1379dc8d65" Jul 24 22:17:58.883305 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.883284 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:58.887586 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.887543 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:58.889525 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.889508 2576 scope.go:117] "RemoveContainer" containerID="a5f88e3f2f2f2693eb176736c113c3d2e65ed5c6ddd2a5b16ff0826393b49b69" Jul 24 22:17:58.895860 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.895839 2576 scope.go:117] "RemoveContainer" containerID="d3f5454f41066a17950bb915b08f3b4c8dc8a08b4c768c26908614c5ebedf9d6" Jul 24 22:17:58.902650 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.902635 2576 scope.go:117] "RemoveContainer" containerID="55285098b83350d04c11e43090eae78d5397465a9ba9ed5fdd84993135b4e9d2" Jul 24 22:17:58.914981 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.914962 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:58.915277 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915265 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="init-config-reloader" Jul 24 22:17:58.915317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915280 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="init-config-reloader" Jul 24 22:17:58.915317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915289 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="config-reloader" Jul 24 22:17:58.915317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915297 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="config-reloader" Jul 24 22:17:58.915317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915309 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy" Jul 24 22:17:58.915317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915316 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915325 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="prometheus" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915330 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="prometheus" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915341 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="thanos-sidecar" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915345 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="thanos-sidecar" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915350 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-web" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915355 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-web" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915361 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc39537f-3022-4277-aabd-60bff510e2bb" containerName="registry" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915366 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc39537f-3022-4277-aabd-60bff510e2bb" containerName="registry" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915373 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-thanos" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915383 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-thanos" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915436 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="prometheus" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915443 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915449 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="config-reloader" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915454 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc39537f-3022-4277-aabd-60bff510e2bb" containerName="registry" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="thanos-sidecar" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915472 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-thanos" Jul 24 22:17:58.915574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.915479 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" containerName="kube-rbac-proxy-web" Jul 24 22:17:58.919291 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.919277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:58.921591 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.921568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 22:17:58.921700 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.921600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 22:17:58.921700 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.921618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 22:17:58.922742 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.922725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4cfl49k0v7hsq\"" Jul 24 22:17:58.922742 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.922739 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 22:17:58.923805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.923784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 22:17:58.923805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.923799 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 22:17:58.924214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.923837 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 22:17:58.924214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.923794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 22:17:58.924214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.923886 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 22:17:58.924214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.924022 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n96r2\"" Jul 24 22:17:58.924339 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.924289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 22:17:58.924339 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.924295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 22:17:58.924339 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.924328 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 22:17:58.931960 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.927779 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 22:17:58.931960 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.931147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 22:17:58.936340 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:58.936322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:59.061007 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.060974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-config-out\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-web-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061489 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdt7k\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-kube-api-access-hdt7k\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061784 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.061784 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.061570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.162773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-config-out\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.162773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.162773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-web-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.162773 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdt7k\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-kube-api-access-hdt7k\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.162997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.163983 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.163902 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.164536 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.164502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.164881 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.164855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.165902 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.165872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-config-out\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.166317 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.166294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.166402 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.166273 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.167523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.166842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.167523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.167317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.167523 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.167365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.167667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c112db9d-4fc7-4532-bbbd-eda08af07e92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168413 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-web-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.168799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.169084 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.168992 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.169455 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.169430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-config\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.169918 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.169892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.170096 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.170080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c112db9d-4fc7-4532-bbbd-eda08af07e92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.171109 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.171088 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdt7k\" (UniqueName: \"kubernetes.io/projected/c112db9d-4fc7-4532-bbbd-eda08af07e92-kube-api-access-hdt7k\") pod \"prometheus-k8s-0\" (UID: \"c112db9d-4fc7-4532-bbbd-eda08af07e92\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.233946 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.233905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:17:59.371334 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.371240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:17:59.373905 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:17:59.373882 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc112db9d_4fc7_4532_bbbd_eda08af07e92.slice/crio-e84c0c906eb27ef2ec9794f3e6f983572c4b2783ad5ca00c0b859e45a7e4866d WatchSource:0}: Error finding container e84c0c906eb27ef2ec9794f3e6f983572c4b2783ad5ca00c0b859e45a7e4866d: Status 404 returned error can't find the container with id e84c0c906eb27ef2ec9794f3e6f983572c4b2783ad5ca00c0b859e45a7e4866d Jul 24 22:17:59.864750 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.864720 2576 generic.go:356] "Generic (PLEG): container finished" podID="c112db9d-4fc7-4532-bbbd-eda08af07e92" containerID="4f622063287e5741e5b42e07fa2d5dd64732e0b0d0af3dbb01805219fd01d24b" exitCode=0 Jul 24 22:17:59.865205 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.864814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerDied","Data":"4f622063287e5741e5b42e07fa2d5dd64732e0b0d0af3dbb01805219fd01d24b"} Jul 24 22:17:59.865205 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.864907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"e84c0c906eb27ef2ec9794f3e6f983572c4b2783ad5ca00c0b859e45a7e4866d"} Jul 24 22:17:59.889410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:17:59.889379 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efd543c-eb65-4464-95d5-374b6baf01dc" path="/var/lib/kubelet/pods/0efd543c-eb65-4464-95d5-374b6baf01dc/volumes" Jul 24 22:18:00.877480 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"2eadd2fe888aaa5f6bc545710d7312384ed39178fb94f343f83503b58e31e3bc"} Jul 24 22:18:00.877864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"91793cb233cc251529f7380b7abb35308988e6295c0076a8037b12751a05b447"} Jul 24 22:18:00.877864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877581 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"5ae54739fa5096a63261d534f4884a4b8d851f70c332633ce137352484ee1a01"} Jul 24 22:18:00.877864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"1e107d8d1104160b1ccbfc2a4c087bd0a30964b4a8983c84511b95fb86578072"} Jul 24 22:18:00.877864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"5bf612e0ec16e5da29d2b68c7f7406493c3bad65c5e9b5305d9d32a09a8c7c19"} Jul 24 22:18:00.877864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.877649 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c112db9d-4fc7-4532-bbbd-eda08af07e92","Type":"ContainerStarted","Data":"4fe6c5bff5738a45bb2dca06b77ca1374dea58da90f9f8cf175297b3828b0236"} Jul 24 22:18:00.906870 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:00.906795 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.88685413 podStartE2EDuration="2.906774701s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.920571ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:18:00.905077728 +0000 UTC m=+203.572088549" watchObservedRunningTime="2026-07-24 22:18:00.906774701 +0000 UTC m=+203.573785523" Jul 24 22:18:04.234200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:04.234164 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:18:59.235101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:59.235011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:18:59.250757 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:18:59.250729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:19:00.112981 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:19:00.112956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:19:37.775548 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:19:37.775520 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 22:21:06.707951 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.707907 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-99577"] Jul 24 22:21:06.710482 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.710459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.712581 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.712504 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v4rcb\"" Jul 24 22:21:06.714498 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.714474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 22:21:06.714498 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.714496 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 22:21:06.714672 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.714618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 22:21:06.714905 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.714888 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 22:21:06.758595 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.758543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-99577"] Jul 24 22:21:06.770106 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.770077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.770264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.770137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.770264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.770201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chtv\" (UniqueName: \"kubernetes.io/projected/f7df0e12-4115-4f18-aa32-98434430ba37-kube-api-access-7chtv\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.770264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.770242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.770374 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.770268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.871256 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.871437 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.871437 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chtv\" (UniqueName: \"kubernetes.io/projected/f7df0e12-4115-4f18-aa32-98434430ba37-kube-api-access-7chtv\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.871437 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.871437 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.877464 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.871931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-manifests\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.877464 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.874793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.878045 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.878015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7df0e12-4115-4f18-aa32-98434430ba37-webhook-cert\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.879754 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.879727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7df0e12-4115-4f18-aa32-98434430ba37-tmp\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:06.881811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:06.881793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chtv\" (UniqueName: \"kubernetes.io/projected/f7df0e12-4115-4f18-aa32-98434430ba37-kube-api-access-7chtv\") pod \"opendatahub-operator-controller-manager-854fb8676-99577\" (UID: \"f7df0e12-4115-4f18-aa32-98434430ba37\") " pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:07.021205 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:07.021119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:07.165098 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:07.165069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854fb8676-99577"] Jul 24 22:21:07.168095 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:21:07.168061 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7df0e12_4115_4f18_aa32_98434430ba37.slice/crio-284884f26cf5ff52aeb0e6619c2bb1d770083dbab93163d8b930ca6470f0baf1 WatchSource:0}: Error finding container 284884f26cf5ff52aeb0e6619c2bb1d770083dbab93163d8b930ca6470f0baf1: Status 404 returned error can't find the container with id 284884f26cf5ff52aeb0e6619c2bb1d770083dbab93163d8b930ca6470f0baf1 Jul 24 22:21:07.169925 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:07.169859 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:21:07.582241 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:07.582198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" event={"ID":"f7df0e12-4115-4f18-aa32-98434430ba37","Type":"ContainerStarted","Data":"284884f26cf5ff52aeb0e6619c2bb1d770083dbab93163d8b930ca6470f0baf1"} Jul 24 22:21:10.598522 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:10.598426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" event={"ID":"f7df0e12-4115-4f18-aa32-98434430ba37","Type":"ContainerStarted","Data":"fa324494c748155aeac4bde3d29df6e2d9b2c2beea603301e092cf3c89e36544"} Jul 24 22:21:11.603880 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:11.603843 2576 generic.go:356] "Generic (PLEG): container finished" podID="f7df0e12-4115-4f18-aa32-98434430ba37" containerID="fa324494c748155aeac4bde3d29df6e2d9b2c2beea603301e092cf3c89e36544" exitCode=0 Jul 24 22:21:11.604239 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:11.603933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" event={"ID":"f7df0e12-4115-4f18-aa32-98434430ba37","Type":"ContainerDied","Data":"fa324494c748155aeac4bde3d29df6e2d9b2c2beea603301e092cf3c89e36544"} Jul 24 22:21:12.610195 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:12.610153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" event={"ID":"f7df0e12-4115-4f18-aa32-98434430ba37","Type":"ContainerStarted","Data":"4bda4ab7a86799aa2a959d24653c68517a332b563dcb7e6b6bd4a945e77bfa7e"} Jul 24 22:21:12.610630 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:12.610279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:12.632714 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:12.632662 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" podStartSLOduration=2.255143635 podStartE2EDuration="6.632646895s" totalImagesPullingTime="3.371760984s" totalInitContainerRuntime="1.005742276s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:12.631482624 +0000 UTC m=+395.298493453" watchObservedRunningTime="2026-07-24 22:21:12.632646895 +0000 UTC m=+395.299657767" Jul 24 22:21:23.617571 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:23.617516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-854fb8676-99577" Jul 24 22:21:25.801160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.801133 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b"] Jul 24 22:21:25.811018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.810996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:25.813289 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.813256 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:21:25.814075 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.814054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 22:21:25.814189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.814073 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jcnrj\"" Jul 24 22:21:25.814189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.814074 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 22:21:25.814189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.814121 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 22:21:25.814189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.814056 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 22:21:25.821614 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.821593 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b"] Jul 24 22:21:25.937418 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.937381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxg4k\" (UniqueName: \"kubernetes.io/projected/29a00523-947f-45ae-92b5-98d7e3ef25c0-kube-api-access-fxg4k\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:25.937608 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.937433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29a00523-947f-45ae-92b5-98d7e3ef25c0-manager-config\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:25.937608 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.937564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:25.937689 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.937616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-metrics-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:25.946912 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.946890 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-c45m2"] Jul 24 22:21:25.949318 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.949295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:25.951369 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.951335 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 22:21:25.951465 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.951414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 22:21:25.951648 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.951634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vqp4s\"" Jul 24 22:21:25.951722 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.951657 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 22:21:25.951948 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.951928 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 22:21:25.966248 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:25.966222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-c45m2"] Jul 24 22:21:26.038034 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29a00523-947f-45ae-92b5-98d7e3ef25c0-manager-config\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.038188 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbad459a-837d-46ed-875b-c241c1d64072-tls-certs\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.038188 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.038254 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-metrics-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.038254 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbad459a-837d-46ed-875b-c241c1d64072-tmp\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.038321 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntfd\" (UniqueName: \"kubernetes.io/projected/bbad459a-837d-46ed-875b-c241c1d64072-kube-api-access-jntfd\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.038321 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxg4k\" (UniqueName: \"kubernetes.io/projected/29a00523-947f-45ae-92b5-98d7e3ef25c0-kube-api-access-fxg4k\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.038779 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.038753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/29a00523-947f-45ae-92b5-98d7e3ef25c0-manager-config\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.040546 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.040526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.040754 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.040733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/29a00523-947f-45ae-92b5-98d7e3ef25c0-metrics-cert\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.050109 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.050091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxg4k\" (UniqueName: \"kubernetes.io/projected/29a00523-947f-45ae-92b5-98d7e3ef25c0-kube-api-access-fxg4k\") pod \"lws-controller-manager-7b958b6f7b-2dx7b\" (UID: \"29a00523-947f-45ae-92b5-98d7e3ef25c0\") " pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.121004 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.120973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:26.138934 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.138910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbad459a-837d-46ed-875b-c241c1d64072-tls-certs\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.139065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.138972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbad459a-837d-46ed-875b-c241c1d64072-tmp\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.139065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.139004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntfd\" (UniqueName: \"kubernetes.io/projected/bbad459a-837d-46ed-875b-c241c1d64072-kube-api-access-jntfd\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.141605 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.141574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbad459a-837d-46ed-875b-c241c1d64072-tls-certs\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.141705 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.141583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbad459a-837d-46ed-875b-c241c1d64072-tmp\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.152160 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.152110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntfd\" (UniqueName: \"kubernetes.io/projected/bbad459a-837d-46ed-875b-c241c1d64072-kube-api-access-jntfd\") pod \"kube-auth-proxy-7d58b94969-c45m2\" (UID: \"bbad459a-837d-46ed-875b-c241c1d64072\") " pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.256524 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.256470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b"] Jul 24 22:21:26.259225 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:21:26.259198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a00523_947f_45ae_92b5_98d7e3ef25c0.slice/crio-7b1f9d0643bcf1e2bbb4156838e409886f27de1935f64954ec6b9d19860db61a WatchSource:0}: Error finding container 7b1f9d0643bcf1e2bbb4156838e409886f27de1935f64954ec6b9d19860db61a: Status 404 returned error can't find the container with id 7b1f9d0643bcf1e2bbb4156838e409886f27de1935f64954ec6b9d19860db61a Jul 24 22:21:26.264495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.264474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" Jul 24 22:21:26.400027 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.400002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58b94969-c45m2"] Jul 24 22:21:26.402200 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:21:26.402178 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbad459a_837d_46ed_875b_c241c1d64072.slice/crio-262165b3e35f1ed8e2fb381f23e3843f71367b745898032a285368441cb03285 WatchSource:0}: Error finding container 262165b3e35f1ed8e2fb381f23e3843f71367b745898032a285368441cb03285: Status 404 returned error can't find the container with id 262165b3e35f1ed8e2fb381f23e3843f71367b745898032a285368441cb03285 Jul 24 22:21:26.669671 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.669586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" event={"ID":"bbad459a-837d-46ed-875b-c241c1d64072","Type":"ContainerStarted","Data":"262165b3e35f1ed8e2fb381f23e3843f71367b745898032a285368441cb03285"} Jul 24 22:21:26.670783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:26.670754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" event={"ID":"29a00523-947f-45ae-92b5-98d7e3ef25c0","Type":"ContainerStarted","Data":"7b1f9d0643bcf1e2bbb4156838e409886f27de1935f64954ec6b9d19860db61a"} Jul 24 22:21:29.685293 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:29.685259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" event={"ID":"29a00523-947f-45ae-92b5-98d7e3ef25c0","Type":"ContainerStarted","Data":"928465ba46adbfb174b19596facd3af6a763eaa59213d6ab73375fc0ae8ebf73"} Jul 24 22:21:29.685771 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:29.685409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:29.686880 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:29.686860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" event={"ID":"bbad459a-837d-46ed-875b-c241c1d64072","Type":"ContainerStarted","Data":"4550e0de24f1ec773c2a8c08d597e33c5f0263fda19e3fa8df1eafb68bfe90ef"} Jul 24 22:21:29.704297 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:29.704251 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" podStartSLOduration=1.485763922 podStartE2EDuration="4.704235264s" totalImagesPullingTime="3.218471342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:29.703374128 +0000 UTC m=+412.370384951" watchObservedRunningTime="2026-07-24 22:21:29.704235264 +0000 UTC m=+412.371246085" Jul 24 22:21:29.718749 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:29.718684 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d58b94969-c45m2" podStartSLOduration=1.6408874020000002 podStartE2EDuration="4.718663307s" totalImagesPullingTime="3.077775905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:29.717787589 +0000 UTC m=+412.384798412" watchObservedRunningTime="2026-07-24 22:21:29.718663307 +0000 UTC m=+412.385674130" Jul 24 22:21:36.281419 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.281378 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln"] Jul 24 22:21:36.285129 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.285105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.287799 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.287778 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dpfq8\"" Jul 24 22:21:36.287915 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.287782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 22:21:36.314408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.314329 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln"] Jul 24 22:21:36.436198 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.436152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.436396 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.436223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7lk9\" (UniqueName: \"kubernetes.io/projected/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-kube-api-access-t7lk9\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.537234 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.537145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7lk9\" (UniqueName: \"kubernetes.io/projected/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-kube-api-access-t7lk9\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.537234 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.537233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.539791 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.539766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.546308 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.546280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7lk9\" (UniqueName: \"kubernetes.io/projected/f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e-kube-api-access-t7lk9\") pod \"llmisvc-controller-manager-777c6bd7d5-rlqln\" (UID: \"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.597077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.597046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:36.737141 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:36.737117 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln"] Jul 24 22:21:36.739901 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:21:36.739875 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf5f1ade1_5d6d_463f_a9f9_0c9fa7e9be7e.slice/crio-b948ba740aebde299925e8c780bf819244a6e9b5ee6b955508aa4a98882a2a3b WatchSource:0}: Error finding container b948ba740aebde299925e8c780bf819244a6e9b5ee6b955508aa4a98882a2a3b: Status 404 returned error can't find the container with id b948ba740aebde299925e8c780bf819244a6e9b5ee6b955508aa4a98882a2a3b Jul 24 22:21:37.726267 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:37.726133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" event={"ID":"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e","Type":"ContainerStarted","Data":"b948ba740aebde299925e8c780bf819244a6e9b5ee6b955508aa4a98882a2a3b"} Jul 24 22:21:40.696932 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:40.696905 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b958b6f7b-2dx7b" Jul 24 22:21:40.740788 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:40.740757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" event={"ID":"f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e","Type":"ContainerStarted","Data":"344abf792c65118de847004ca6307a39edbd64ca7386b21d750c0ff07c86dc6f"} Jul 24 22:21:40.740962 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:40.740861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:21:40.756971 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:21:40.756922 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" podStartSLOduration=1.837836232 podStartE2EDuration="4.756907192s" totalImagesPullingTime="2.91907096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:21:40.75607685 +0000 UTC m=+423.423087671" watchObservedRunningTime="2026-07-24 22:21:40.756907192 +0000 UTC m=+423.423918013" Jul 24 22:22:11.747451 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:22:11.747379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rlqln" Jul 24 22:23:19.524924 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.524889 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2"] Jul 24 22:23:19.528218 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.528200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.530745 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.530723 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b4g4c\"" Jul 24 22:23:19.531148 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.531125 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 22:23:19.531251 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.531176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 22:23:19.545151 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.545132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2"] Jul 24 22:23:19.636983 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.636945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rldw\" (UniqueName: \"kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.637139 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.636988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.738206 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.738160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rldw\" (UniqueName: \"kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.738206 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.738211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.738626 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.738607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.751864 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.751833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rldw\" (UniqueName: \"kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw\") pod \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.837709 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.837621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:19.971411 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:19.971381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2"] Jul 24 22:23:19.974612 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:23:19.974582 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10537b91_3fe9_48bc_a606_94e3374ef24f.slice/crio-1330b2c272471b8289f3a9dddcfd9d4e76816a8479587dc4ef2ee6c3a13719c4 WatchSource:0}: Error finding container 1330b2c272471b8289f3a9dddcfd9d4e76816a8479587dc4ef2ee6c3a13719c4: Status 404 returned error can't find the container with id 1330b2c272471b8289f3a9dddcfd9d4e76816a8479587dc4ef2ee6c3a13719c4 Jul 24 22:23:20.035801 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.035764 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2"] Jul 24 22:23:20.041611 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.041586 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2"] Jul 24 22:23:20.060601 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.060575 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:20.064972 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.064954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.077470 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.077447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:20.142087 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.142056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.142087 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.142098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwdq\" (UniqueName: \"kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.243505 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.243475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.243702 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.243518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwdq\" (UniqueName: \"kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.243904 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.243885 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.252667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.252622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwdq\" (UniqueName: \"kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq\") pod \"kuadrant-operator-controller-manager-55467d9f7-8ztqb\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.378077 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.378048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:20.528357 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:20.528322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:20.551212 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:23:20.551181 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d845dd_87cc_45a4_86f1_e605c7d06438.slice/crio-2bc31c5546ddfd57a25fc2c68bc68d8f570cffc8aff63d2c911930c461f9f2ff WatchSource:0}: Error finding container 2bc31c5546ddfd57a25fc2c68bc68d8f570cffc8aff63d2c911930c461f9f2ff: Status 404 returned error can't find the container with id 2bc31c5546ddfd57a25fc2c68bc68d8f570cffc8aff63d2c911930c461f9f2ff Jul 24 22:23:21.173623 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:21.173589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" event={"ID":"f6d845dd-87cc-45a4-86f1-e605c7d06438","Type":"ContainerStarted","Data":"2bc31c5546ddfd57a25fc2c68bc68d8f570cffc8aff63d2c911930c461f9f2ff"} Jul 24 22:23:24.195688 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.195651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" event={"ID":"f6d845dd-87cc-45a4-86f1-e605c7d06438","Type":"ContainerStarted","Data":"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b"} Jul 24 22:23:24.196133 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.195775 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:24.197395 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.197351 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" containerName="manager" containerID="cri-o://43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f" gracePeriod=2 Jul 24 22:23:24.213391 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.213297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" podStartSLOduration=0.79666207 podStartE2EDuration="4.213279632s" totalImagesPullingTime="3.416617562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:24.212074765 +0000 UTC m=+526.879085588" watchObservedRunningTime="2026-07-24 22:23:24.213279632 +0000 UTC m=+526.880290454" Jul 24 22:23:24.442983 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.442955 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:24.444695 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:23:24.444672 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" is forbidden: User \"system:node:ip-10-0-132-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-132.ec2.internal' and this object" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:24.584398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.584302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume\") pod \"10537b91-3fe9-48bc-a606-94e3374ef24f\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " Jul 24 22:23:24.584398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.584396 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rldw\" (UniqueName: \"kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw\") pod \"10537b91-3fe9-48bc-a606-94e3374ef24f\" (UID: \"10537b91-3fe9-48bc-a606-94e3374ef24f\") " Jul 24 22:23:24.584652 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.584601 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10537b91-3fe9-48bc-a606-94e3374ef24f" (UID: "10537b91-3fe9-48bc-a606-94e3374ef24f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:23:24.584706 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.584681 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10537b91-3fe9-48bc-a606-94e3374ef24f-extensions-socket-volume\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:23:24.586709 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.586679 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw" (OuterVolumeSpecName: "kube-api-access-4rldw") pod "10537b91-3fe9-48bc-a606-94e3374ef24f" (UID: "10537b91-3fe9-48bc-a606-94e3374ef24f"). InnerVolumeSpecName "kube-api-access-4rldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:23:24.685439 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:24.685389 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rldw\" (UniqueName: \"kubernetes.io/projected/10537b91-3fe9-48bc-a606-94e3374ef24f-kube-api-access-4rldw\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:23:25.203837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.203799 2576 generic.go:356] "Generic (PLEG): container finished" podID="10537b91-3fe9-48bc-a606-94e3374ef24f" containerID="43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f" exitCode=2 Jul 24 22:23:25.203837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.203843 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:25.204376 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.203900 2576 scope.go:117] "RemoveContainer" containerID="43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f" Jul 24 22:23:25.205877 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:23:25.205846 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" is forbidden: User \"system:node:ip-10-0-132-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-132.ec2.internal' and this object" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:25.212903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.212881 2576 scope.go:117] "RemoveContainer" containerID="43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f" Jul 24 22:23:25.213170 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:23:25.213151 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f\": container with ID starting with 43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f not found: ID does not exist" containerID="43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f" Jul 24 22:23:25.213242 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.213178 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f"} err="failed to get container status \"43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f\": rpc error: code = NotFound desc = could not find container \"43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f\": container with ID starting with 43cf03baf426c4713cb2733b4afa039bfaa0a6b32c19f0f9feed7f386e1c388f not found: ID does not exist" Jul 24 22:23:25.213348 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:23:25.213333 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dstg2\" is forbidden: User \"system:node:ip-10-0-132-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-132.ec2.internal' and this object" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dstg2" Jul 24 22:23:25.890126 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:25.890080 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" path="/var/lib/kubelet/pods/10537b91-3fe9-48bc-a606-94e3374ef24f/volumes" Jul 24 22:23:35.205909 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:35.205876 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:47.569283 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.569245 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:47.569732 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.569469 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" podUID="f6d845dd-87cc-45a4-86f1-e605c7d06438" containerName="manager" containerID="cri-o://5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b" gracePeriod=10 Jul 24 22:23:47.831258 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.831201 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:47.887073 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.887037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dwdq\" (UniqueName: \"kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq\") pod \"f6d845dd-87cc-45a4-86f1-e605c7d06438\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " Jul 24 22:23:47.887391 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.887374 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume\") pod \"f6d845dd-87cc-45a4-86f1-e605c7d06438\" (UID: \"f6d845dd-87cc-45a4-86f1-e605c7d06438\") " Jul 24 22:23:47.888324 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.888292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f6d845dd-87cc-45a4-86f1-e605c7d06438" (UID: "f6d845dd-87cc-45a4-86f1-e605c7d06438"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:23:47.894098 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.894002 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq" (OuterVolumeSpecName: "kube-api-access-8dwdq") pod "f6d845dd-87cc-45a4-86f1-e605c7d06438" (UID: "f6d845dd-87cc-45a4-86f1-e605c7d06438"). InnerVolumeSpecName "kube-api-access-8dwdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:23:47.988765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.988726 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwdq\" (UniqueName: \"kubernetes.io/projected/f6d845dd-87cc-45a4-86f1-e605c7d06438-kube-api-access-8dwdq\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:23:47.988765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:47.988759 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f6d845dd-87cc-45a4-86f1-e605c7d06438-extensions-socket-volume\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:23:48.308621 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.308580 2576 generic.go:356] "Generic (PLEG): container finished" podID="f6d845dd-87cc-45a4-86f1-e605c7d06438" containerID="5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b" exitCode=0 Jul 24 22:23:48.308821 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.308639 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" Jul 24 22:23:48.308821 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.308642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" event={"ID":"f6d845dd-87cc-45a4-86f1-e605c7d06438","Type":"ContainerDied","Data":"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b"} Jul 24 22:23:48.308821 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.308697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb" event={"ID":"f6d845dd-87cc-45a4-86f1-e605c7d06438","Type":"ContainerDied","Data":"2bc31c5546ddfd57a25fc2c68bc68d8f570cffc8aff63d2c911930c461f9f2ff"} Jul 24 22:23:48.308821 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.308719 2576 scope.go:117] "RemoveContainer" containerID="5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b" Jul 24 22:23:48.317627 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.317609 2576 scope.go:117] "RemoveContainer" containerID="5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b" Jul 24 22:23:48.317901 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:23:48.317884 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b\": container with ID starting with 5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b not found: ID does not exist" containerID="5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b" Jul 24 22:23:48.317940 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.317911 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b"} err="failed to get container status \"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b\": rpc error: code = NotFound desc = could not find container \"5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b\": container with ID starting with 5d83f53f12ff380da4634844802bef3831058453f4ee8b1b77208259e17ffb3b not found: ID does not exist" Jul 24 22:23:48.329064 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.329037 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:48.332797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:48.332773 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8ztqb"] Jul 24 22:23:49.890142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:49.890109 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d845dd-87cc-45a4-86f1-e605c7d06438" path="/var/lib/kubelet/pods/f6d845dd-87cc-45a4-86f1-e605c7d06438/volumes" Jul 24 22:23:51.558622 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.558586 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5"] Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.558917 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" containerName="manager" Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.558927 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" containerName="manager" Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.558947 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d845dd-87cc-45a4-86f1-e605c7d06438" containerName="manager" Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.558953 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d845dd-87cc-45a4-86f1-e605c7d06438" containerName="manager" Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.559005 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="10537b91-3fe9-48bc-a606-94e3374ef24f" containerName="manager" Jul 24 22:23:51.559018 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.559015 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d845dd-87cc-45a4-86f1-e605c7d06438" containerName="manager" Jul 24 22:23:51.565060 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.565038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.567014 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.566985 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 22:23:51.567155 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.567086 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8hlnd\"" Jul 24 22:23:51.574648 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.574624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5"] Jul 24 22:23:51.621772 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.621772 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab261fc0-d310-4080-9800-49de253bc260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glkb\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-kube-api-access-7glkb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.621986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622042 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.622037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab261fc0-d310-4080-9800-49de253bc260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.622351 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.622117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723197 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723156 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723197 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab261fc0-d310-4080-9800-49de253bc260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glkb\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-kube-api-access-7glkb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab261fc0-d310-4080-9800-49de253bc260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.723815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.723734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.724020 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.724000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.724086 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.724041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.724274 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.724257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab261fc0-d310-4080-9800-49de253bc260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.725990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.725973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab261fc0-d310-4080-9800-49de253bc260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.726058 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.726001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab261fc0-d310-4080-9800-49de253bc260-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.730915 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.730883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.731040 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.730956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glkb\" (UniqueName: \"kubernetes.io/projected/ab261fc0-d310-4080-9800-49de253bc260-kube-api-access-7glkb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4xpv5\" (UID: \"ab261fc0-d310-4080-9800-49de253bc260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:51.877627 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:51.877590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:52.021252 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:52.021229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5"] Jul 24 22:23:52.024435 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:23:52.024399 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab261fc0_d310_4080_9800_49de253bc260.slice/crio-e7a18eaa1ba92ecbd69fb64ef8ee43213680d3795f572ebd22cd9c146740fc67 WatchSource:0}: Error finding container e7a18eaa1ba92ecbd69fb64ef8ee43213680d3795f572ebd22cd9c146740fc67: Status 404 returned error can't find the container with id e7a18eaa1ba92ecbd69fb64ef8ee43213680d3795f572ebd22cd9c146740fc67 Jul 24 22:23:52.329284 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:52.329199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" event={"ID":"ab261fc0-d310-4080-9800-49de253bc260","Type":"ContainerStarted","Data":"e7a18eaa1ba92ecbd69fb64ef8ee43213680d3795f572ebd22cd9c146740fc67"} Jul 24 22:23:54.754896 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:54.754859 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:23:54.755200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:54.754935 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:23:54.755200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:54.754967 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:23:55.345167 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:55.345132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" event={"ID":"ab261fc0-d310-4080-9800-49de253bc260","Type":"ContainerStarted","Data":"24c78b4d754a084fcb9171ff7f55cfcf724e3a9d25d8a108cc61927fad739cf6"} Jul 24 22:23:55.364973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:55.364930 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podStartSLOduration=1.637190678 podStartE2EDuration="4.364915189s" totalImagesPullingTime="2.727724511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:23:55.363379136 +0000 UTC m=+558.030389957" watchObservedRunningTime="2026-07-24 22:23:55.364915189 +0000 UTC m=+558.031926010" Jul 24 22:23:55.878089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:55.878051 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:55.882939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:55.882913 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:56.349473 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:56.349380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:23:56.350513 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:23:56.350492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:24:06.506284 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.506246 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:24:06.509750 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.509734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.511817 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.511792 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 22:24:06.511946 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.511796 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 22:24:06.512343 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.512325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 22:24:06.512396 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.512380 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bckgb\"" Jul 24 22:24:06.518584 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.518544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:24:06.547755 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.547731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:24:06.657621 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.657544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/084fe24f-935e-44a4-a254-c60909f6af3d-kube-api-access-zfmjm\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.657813 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.657634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/084fe24f-935e-44a4-a254-c60909f6af3d-config-file\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.759078 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.758992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/084fe24f-935e-44a4-a254-c60909f6af3d-kube-api-access-zfmjm\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.759078 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.759035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/084fe24f-935e-44a4-a254-c60909f6af3d-config-file\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.759632 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.759613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/084fe24f-935e-44a4-a254-c60909f6af3d-config-file\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.766729 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.766706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/084fe24f-935e-44a4-a254-c60909f6af3d-kube-api-access-zfmjm\") pod \"limitador-limitador-54d6d988d5-8c565\" (UID: \"084fe24f-935e-44a4-a254-c60909f6af3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.821743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.821718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:06.952697 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.952612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:24:06.955734 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:24:06.955704 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084fe24f_935e_44a4_a254_c60909f6af3d.slice/crio-8b33f8dcb7d9269078077f69278ee5378b04c985b9f8346fd9e065a403ca1f62 WatchSource:0}: Error finding container 8b33f8dcb7d9269078077f69278ee5378b04c985b9f8346fd9e065a403ca1f62: Status 404 returned error can't find the container with id 8b33f8dcb7d9269078077f69278ee5378b04c985b9f8346fd9e065a403ca1f62 Jul 24 22:24:06.968196 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.968175 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:06.972634 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.972615 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:06.975065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.975049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j8mhc\"" Jul 24 22:24:06.979526 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:06.979501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:07.061893 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.061826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bsb\" (UniqueName: \"kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb\") pod \"authorino-58cd955675-84mmw\" (UID: \"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497\") " pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:07.163230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.163186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bsb\" (UniqueName: \"kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb\") pod \"authorino-58cd955675-84mmw\" (UID: \"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497\") " pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:07.172609 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.172577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bsb\" (UniqueName: \"kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb\") pod \"authorino-58cd955675-84mmw\" (UID: \"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497\") " pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:07.294591 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.294535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:07.401382 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.401343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" event={"ID":"084fe24f-935e-44a4-a254-c60909f6af3d","Type":"ContainerStarted","Data":"8b33f8dcb7d9269078077f69278ee5378b04c985b9f8346fd9e065a403ca1f62"} Jul 24 22:24:07.442637 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:07.442589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:07.445219 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:24:07.445187 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b2ceaa_ba79_4d5a_bd64_1bb835e6e497.slice/crio-6d66537ca5c54725206dd23f7e1d5eeb1129c7ea293ef33a02c947b847fb0dc6 WatchSource:0}: Error finding container 6d66537ca5c54725206dd23f7e1d5eeb1129c7ea293ef33a02c947b847fb0dc6: Status 404 returned error can't find the container with id 6d66537ca5c54725206dd23f7e1d5eeb1129c7ea293ef33a02c947b847fb0dc6 Jul 24 22:24:08.409933 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:08.409889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84mmw" event={"ID":"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497","Type":"ContainerStarted","Data":"6d66537ca5c54725206dd23f7e1d5eeb1129c7ea293ef33a02c947b847fb0dc6"} Jul 24 22:24:11.425946 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.425902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84mmw" event={"ID":"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497","Type":"ContainerStarted","Data":"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b"} Jul 24 22:24:11.427634 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.427606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" event={"ID":"084fe24f-935e-44a4-a254-c60909f6af3d","Type":"ContainerStarted","Data":"0be726f0eb4ba01215341ee71301ef208c6266183ca4d2961ac7504cb2c48823"} Jul 24 22:24:11.427735 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.427664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:24:11.446352 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.446306 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-84mmw" podStartSLOduration=1.55931257 podStartE2EDuration="5.446290075s" totalImagesPullingTime="3.886977505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:11.440326558 +0000 UTC m=+574.107337381" watchObservedRunningTime="2026-07-24 22:24:11.446290075 +0000 UTC m=+574.113300897" Jul 24 22:24:11.459363 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.459311 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" podStartSLOduration=1.08908534 podStartE2EDuration="5.459294753s" totalImagesPullingTime="4.370209413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:11.457908911 +0000 UTC m=+574.124919736" watchObservedRunningTime="2026-07-24 22:24:11.459294753 +0000 UTC m=+574.126305574" Jul 24 22:24:11.745182 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:11.745148 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:13.437209 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:13.437169 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-84mmw" podUID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" containerName="authorino" containerID="cri-o://2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b" gracePeriod=30 Jul 24 22:24:13.683316 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:13.683281 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:13.828065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:13.827971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98bsb\" (UniqueName: \"kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb\") pod \"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497\" (UID: \"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497\") " Jul 24 22:24:13.830165 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:13.830137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb" (OuterVolumeSpecName: "kube-api-access-98bsb") pod "70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" (UID: "70b2ceaa-ba79-4d5a-bd64-1bb835e6e497"). InnerVolumeSpecName "kube-api-access-98bsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:24:13.929205 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:13.929165 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98bsb\" (UniqueName: \"kubernetes.io/projected/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497-kube-api-access-98bsb\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:24:14.443254 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.443217 2576 generic.go:356] "Generic (PLEG): container finished" podID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" containerID="2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b" exitCode=0 Jul 24 22:24:14.443254 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.443258 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-84mmw" Jul 24 22:24:14.443778 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.443309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84mmw" event={"ID":"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497","Type":"ContainerDied","Data":"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b"} Jul 24 22:24:14.443778 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.443351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-84mmw" event={"ID":"70b2ceaa-ba79-4d5a-bd64-1bb835e6e497","Type":"ContainerDied","Data":"6d66537ca5c54725206dd23f7e1d5eeb1129c7ea293ef33a02c947b847fb0dc6"} Jul 24 22:24:14.443778 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.443367 2576 scope.go:117] "RemoveContainer" containerID="2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b" Jul 24 22:24:14.451951 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.451934 2576 scope.go:117] "RemoveContainer" containerID="2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b" Jul 24 22:24:14.452219 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:24:14.452201 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b\": container with ID starting with 2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b not found: ID does not exist" containerID="2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b" Jul 24 22:24:14.452283 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.452232 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b"} err="failed to get container status \"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b\": rpc error: code = NotFound desc = could not find container \"2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b\": container with ID starting with 2e6d64b30c9402ae4d0bb2c91ffd0cd98e1a2ddbae38f9005327c03d50648d2b not found: ID does not exist" Jul 24 22:24:14.457952 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.457927 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:14.461014 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:14.460991 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-84mmw"] Jul 24 22:24:15.890083 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:15.890053 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" path="/var/lib/kubelet/pods/70b2ceaa-ba79-4d5a-bd64-1bb835e6e497/volumes" Jul 24 22:24:22.434189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:24:22.434158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8c565" Jul 24 22:25:21.496189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:21.496150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:25:27.390805 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:27.390759 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4xpv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 22:25:27.391216 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:27.390834 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podUID="ab261fc0-d310-4080-9800-49de253bc260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 22:25:29.793090 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.793053 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab261fc0-d310-4080-9800-49de253bc260" containerID="24c78b4d754a084fcb9171ff7f55cfcf724e3a9d25d8a108cc61927fad739cf6" exitCode=0 Jul 24 22:25:29.793532 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.793129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" event={"ID":"ab261fc0-d310-4080-9800-49de253bc260","Type":"ContainerDied","Data":"24c78b4d754a084fcb9171ff7f55cfcf724e3a9d25d8a108cc61927fad739cf6"} Jul 24 22:25:29.793597 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.793587 2576 scope.go:117] "RemoveContainer" containerID="24c78b4d754a084fcb9171ff7f55cfcf724e3a9d25d8a108cc61927fad739cf6" Jul 24 22:25:29.794565 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.794520 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:25:29.794666 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.794620 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:25:29.794730 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:29.794669 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 22:25:30.799734 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:30.799699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" event={"ID":"ab261fc0-d310-4080-9800-49de253bc260","Type":"ContainerStarted","Data":"ce2971ce0ec7e770e3e5110a85bd3bbe73a25f3a101595352bf1566a89c98774"} Jul 24 22:25:30.878628 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:30.878590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:25:30.878920 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:30.878885 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4xpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 24 22:25:30.879038 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:30.878946 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podUID="ab261fc0-d310-4080-9800-49de253bc260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 24 22:25:31.878004 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:31.877967 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4xpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 24 22:25:31.878392 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:31.878036 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podUID="ab261fc0-d310-4080-9800-49de253bc260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 24 22:25:32.878986 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:32.878953 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4xpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 24 22:25:32.879366 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:32.879026 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podUID="ab261fc0-d310-4080-9800-49de253bc260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 24 22:25:33.878974 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:33.878933 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4xpv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 24 22:25:33.879189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:33.878998 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" podUID="ab261fc0-d310-4080-9800-49de253bc260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 24 22:25:34.882086 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:34.882052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:25:34.882468 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:34.882417 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:25:34.883036 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:34.883020 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4xpv5" Jul 24 22:25:37.357409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:37.357379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:25:44.254973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:44.254941 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:25:50.388040 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.388008 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd"] Jul 24 22:25:50.388508 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.388392 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" containerName="authorino" Jul 24 22:25:50.388508 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.388405 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" containerName="authorino" Jul 24 22:25:50.388508 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.388478 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b2ceaa-ba79-4d5a-bd64-1bb835e6e497" containerName="authorino" Jul 24 22:25:50.391774 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.391758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.393794 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.393769 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 22:25:50.393794 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.393784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 22:25:50.394281 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.394264 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v4bdf\"" Jul 24 22:25:50.394394 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.394297 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 22:25:50.399715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.399688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd"] Jul 24 22:25:50.500705 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.500901 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.500901 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d47a34-9a37-447f-8327-7ca622cf14ff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.500901 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.500901 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.501073 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.500919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q72\" (UniqueName: \"kubernetes.io/projected/a7d47a34-9a37-447f-8327-7ca622cf14ff-kube-api-access-29q72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602357 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602521 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q72\" (UniqueName: \"kubernetes.io/projected/a7d47a34-9a37-447f-8327-7ca622cf14ff-kube-api-access-29q72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602521 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602521 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602521 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d47a34-9a37-447f-8327-7ca622cf14ff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602737 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602793 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602846 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.602942 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.602925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.604765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.604740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7d47a34-9a37-447f-8327-7ca622cf14ff-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.605241 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.605217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7d47a34-9a37-447f-8327-7ca622cf14ff-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.609907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.609888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q72\" (UniqueName: \"kubernetes.io/projected/a7d47a34-9a37-447f-8327-7ca622cf14ff-kube-api-access-29q72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggfdd\" (UID: \"a7d47a34-9a37-447f-8327-7ca622cf14ff\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.704088 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.704002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:25:50.843676 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.843651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd"] Jul 24 22:25:50.846188 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:25:50.846156 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d47a34_9a37_447f_8327_7ca622cf14ff.slice/crio-0fbd63b8071b437e6abac19173bc8c671e8f69adbd4db84cb020489e5bac0814 WatchSource:0}: Error finding container 0fbd63b8071b437e6abac19173bc8c671e8f69adbd4db84cb020489e5bac0814: Status 404 returned error can't find the container with id 0fbd63b8071b437e6abac19173bc8c671e8f69adbd4db84cb020489e5bac0814 Jul 24 22:25:50.889834 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:50.889798 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" event={"ID":"a7d47a34-9a37-447f-8327-7ca622cf14ff","Type":"ContainerStarted","Data":"0fbd63b8071b437e6abac19173bc8c671e8f69adbd4db84cb020489e5bac0814"} Jul 24 22:25:51.250621 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:51.250584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:25:52.899868 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:52.899832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" event={"ID":"a7d47a34-9a37-447f-8327-7ca622cf14ff","Type":"ContainerStarted","Data":"d69e5c58d67848f5c155164f8537a9577571343fe02a590481bee36b2834bf20"} Jul 24 22:25:52.917067 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:25:52.917017 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" podStartSLOduration=1.9288669870000001 podStartE2EDuration="2.917002503s" totalImagesPullingTime="988.135516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:52.915111908 +0000 UTC m=+675.582122730" watchObservedRunningTime="2026-07-24 22:25:52.917002503 +0000 UTC m=+675.584013325" Jul 24 22:26:00.704473 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:00.704436 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:26:00.705064 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:00.704500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:26:00.717277 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:00.717252 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:26:00.947320 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:00.947290 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggfdd" Jul 24 22:26:13.952659 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:13.952619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:26:30.695705 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.695626 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh"] Jul 24 22:26:30.699853 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.699830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.701788 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.701763 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 22:26:30.708868 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.708841 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh"] Jul 24 22:26:30.867000 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.866965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.867000 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.867008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.867245 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.867036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.867245 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.867087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pdp\" (UniqueName: \"kubernetes.io/projected/c161d106-1fbf-4da9-ba00-ddbd589a34e8-kube-api-access-29pdp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.867245 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.867152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.867245 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.867183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968343 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968343 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968624 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968624 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968749 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968749 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pdp\" (UniqueName: \"kubernetes.io/projected/c161d106-1fbf-4da9-ba00-ddbd589a34e8-kube-api-access-29pdp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968749 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968905 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.968962 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.968936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.970896 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.970870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c161d106-1fbf-4da9-ba00-ddbd589a34e8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.971122 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.971107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c161d106-1fbf-4da9-ba00-ddbd589a34e8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:30.976520 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:30.976501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pdp\" (UniqueName: \"kubernetes.io/projected/c161d106-1fbf-4da9-ba00-ddbd589a34e8-kube-api-access-29pdp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wbpfh\" (UID: \"c161d106-1fbf-4da9-ba00-ddbd589a34e8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:31.011643 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:31.011600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:31.152934 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:31.152909 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh"] Jul 24 22:26:31.155751 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:26:31.155721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc161d106_1fbf_4da9_ba00_ddbd589a34e8.slice/crio-1ef92204dd6f60f4893a4e7c30566c83815b33abd9a0e58eb45abffa43831f96 WatchSource:0}: Error finding container 1ef92204dd6f60f4893a4e7c30566c83815b33abd9a0e58eb45abffa43831f96: Status 404 returned error can't find the container with id 1ef92204dd6f60f4893a4e7c30566c83815b33abd9a0e58eb45abffa43831f96 Jul 24 22:26:31.157131 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:31.157113 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:26:32.079951 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:32.079915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" event={"ID":"c161d106-1fbf-4da9-ba00-ddbd589a34e8","Type":"ContainerStarted","Data":"4ce701be9f905a91f48d7213f43541aab03395acd9bb82629ac5234bd10dc705"} Jul 24 22:26:32.080417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:32.079972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" event={"ID":"c161d106-1fbf-4da9-ba00-ddbd589a34e8","Type":"ContainerStarted","Data":"1ef92204dd6f60f4893a4e7c30566c83815b33abd9a0e58eb45abffa43831f96"} Jul 24 22:26:32.097014 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:32.096960 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" podStartSLOduration=1.892893608 podStartE2EDuration="2.096942432s" totalImagesPullingTime="204.048824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:26:32.095959834 +0000 UTC m=+714.762970670" watchObservedRunningTime="2026-07-24 22:26:32.096942432 +0000 UTC m=+714.763953254" Jul 24 22:26:33.053139 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:33.053101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:26:34.500806 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.500770 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f"] Jul 24 22:26:34.504610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.504587 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.506652 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.506627 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 22:26:34.512727 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.512701 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f"] Jul 24 22:26:34.603209 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.603367 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2kh\" (UniqueName: \"kubernetes.io/projected/9cd3732c-0934-4301-9657-9ec3029c2b49-kube-api-access-gk2kh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.603367 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.603443 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.603443 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd3732c-0934-4301-9657-9ec3029c2b49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.603443 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.603427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.704871 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.704871 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd3732c-0934-4301-9657-9ec3029c2b49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705135 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705135 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705135 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2kh\" (UniqueName: \"kubernetes.io/projected/9cd3732c-0934-4301-9657-9ec3029c2b49-kube-api-access-gk2kh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705135 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.704985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705384 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.705361 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705463 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.705424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.705514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.705480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.707922 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.707900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd3732c-0934-4301-9657-9ec3029c2b49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.708142 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.708126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd3732c-0934-4301-9657-9ec3029c2b49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.716730 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.716707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2kh\" (UniqueName: \"kubernetes.io/projected/9cd3732c-0934-4301-9657-9ec3029c2b49-kube-api-access-gk2kh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v844f\" (UID: \"9cd3732c-0934-4301-9657-9ec3029c2b49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.825358 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.825265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:34.964934 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:34.964908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f"] Jul 24 22:26:34.967680 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:26:34.967649 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd3732c_0934_4301_9657_9ec3029c2b49.slice/crio-6a632df2bc84994b4dc4358fb82a2d7a08c912518d46cfcb0b5f13a8448aa350 WatchSource:0}: Error finding container 6a632df2bc84994b4dc4358fb82a2d7a08c912518d46cfcb0b5f13a8448aa350: Status 404 returned error can't find the container with id 6a632df2bc84994b4dc4358fb82a2d7a08c912518d46cfcb0b5f13a8448aa350 Jul 24 22:26:35.096441 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:35.096346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" event={"ID":"9cd3732c-0934-4301-9657-9ec3029c2b49","Type":"ContainerStarted","Data":"6a632df2bc84994b4dc4358fb82a2d7a08c912518d46cfcb0b5f13a8448aa350"} Jul 24 22:26:36.103273 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:36.103229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" event={"ID":"9cd3732c-0934-4301-9657-9ec3029c2b49","Type":"ContainerStarted","Data":"b8ed665c29b5d5f01375bfac42495f27a4429fbe6e8e3f3d0b5a05d1fb28a7e0"} Jul 24 22:26:36.119119 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:36.119068 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" podStartSLOduration=1.908175314 podStartE2EDuration="2.119034065s" totalImagesPullingTime="210.858751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:26:36.117892417 +0000 UTC m=+718.784903241" watchObservedRunningTime="2026-07-24 22:26:36.119034065 +0000 UTC m=+718.786044888" Jul 24 22:26:41.012495 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:41.012456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:41.012958 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:41.012524 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:41.024967 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:41.024943 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:41.135748 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:41.135720 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wbpfh" Jul 24 22:26:44.357256 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:44.357217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:26:44.826210 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:44.826113 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:44.826210 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:44.826168 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:44.838483 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:44.838451 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:26:45.154636 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:26:45.154604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v844f" Jul 24 22:28:29.048265 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:28:29.048234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:28:39.044738 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:28:39.044697 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:28:48.448877 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:28:48.448840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:30:29.649703 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:30:29.649665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:30:40.850195 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:30:40.850155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:31:17.745024 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:31:17.744987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:31:26.754911 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:31:26.754872 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:31:37.354563 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:31:37.354528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:31:42.148903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:31:42.148867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:31:53.648820 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:31:53.648790 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:32:03.458200 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:32:03.458163 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:32:14.449649 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:32:14.449613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:32:24.551712 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:32:24.551675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:32:49.870637 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:32:49.870603 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:33:02.150945 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:33:02.150905 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:33:23.254161 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:33:23.254114 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:33:33.466454 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:33:33.466418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:33:49.857484 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:33:49.857422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:33:59.847147 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:33:59.847057 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:34:08.654825 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:34:08.654785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:34:18.052437 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:34:18.052402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:34:27.452452 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:34:27.452412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:36:15.548895 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:36:15.548862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:36:26.152729 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:36:26.152694 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:36:37.749687 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:36:37.749657 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:36:50.943002 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:36:50.942966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:00.640767 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:00.640692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:11.742181 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:11.742151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:20.551502 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:20.551468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:32.148193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:32.148154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:43.148534 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:43.148499 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:37:50.946626 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:37:50.946588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:38:01.749275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:01.749241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:38:25.958280 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:25.958247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:38:33.555964 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:33.555931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:38:34.713967 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.713938 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:38:34.717287 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.717271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.719076 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.719055 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-94df6b07-kserve-self-signed-certs\"" Jul 24 22:38:34.732896 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.732877 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:38:34.853113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.853113 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.853299 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.853299 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.853299 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.853299 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.853295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96k9p\" (UniqueName: \"kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.953722 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.953691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.953722 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.953723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.953949 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.953740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.953949 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.953896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.954065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.954003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.954065 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.954029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96k9p\" (UniqueName: \"kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.954154 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.954098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.954218 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.954153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.954281 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.954257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.956180 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.956160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.956422 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.956407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:34.962478 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:34.962452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96k9p\" (UniqueName: \"kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p\") pod \"e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:35.025983 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:35.025915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:35.159177 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:35.159049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:38:35.161356 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:38:35.161329 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf644919c_eacb_467b_9771_ea6c51bbd43b.slice/crio-6a2eed4c65808936c942c212ac9f2a4d442ffdb3b32c53194b77f6bb81352f9b WatchSource:0}: Error finding container 6a2eed4c65808936c942c212ac9f2a4d442ffdb3b32c53194b77f6bb81352f9b: Status 404 returned error can't find the container with id 6a2eed4c65808936c942c212ac9f2a4d442ffdb3b32c53194b77f6bb81352f9b Jul 24 22:38:35.420398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:35.420367 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" event={"ID":"f644919c-eacb-467b-9771-ea6c51bbd43b","Type":"ContainerStarted","Data":"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba"} Jul 24 22:38:35.420577 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:35.420408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" event={"ID":"f644919c-eacb-467b-9771-ea6c51bbd43b","Type":"ContainerStarted","Data":"6a2eed4c65808936c942c212ac9f2a4d442ffdb3b32c53194b77f6bb81352f9b"} Jul 24 22:38:35.435462 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:35.435418 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" podStartSLOduration=1.435404252 podStartE2EDuration="1.435404252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:38:35.434357663 +0000 UTC m=+1438.101368483" watchObservedRunningTime="2026-07-24 22:38:35.435404252 +0000 UTC m=+1438.102415073" Jul 24 22:38:45.026715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:45.026670 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:45.026715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:45.026727 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:45.038078 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:45.038054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:45.476954 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:45.476932 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:38:45.656019 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:45.655988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:38:56.050547 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:38:56.050507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:39:24.345500 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:24.345466 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:39:28.648405 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:28.648369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:39:39.365030 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.364993 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" podUID="f644919c-eacb-467b-9771-ea6c51bbd43b" containerName="main" containerID="cri-o://1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba" gracePeriod=45 Jul 24 22:39:39.614519 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.614498 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:39:39.712224 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.712153 2576 generic.go:356] "Generic (PLEG): container finished" podID="f644919c-eacb-467b-9771-ea6c51bbd43b" containerID="1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba" exitCode=0 Jul 24 22:39:39.712224 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.712196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" Jul 24 22:39:39.712410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.712246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" event={"ID":"f644919c-eacb-467b-9771-ea6c51bbd43b","Type":"ContainerDied","Data":"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba"} Jul 24 22:39:39.712410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.712289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk" event={"ID":"f644919c-eacb-467b-9771-ea6c51bbd43b","Type":"ContainerDied","Data":"6a2eed4c65808936c942c212ac9f2a4d442ffdb3b32c53194b77f6bb81352f9b"} Jul 24 22:39:39.712410 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.712311 2576 scope.go:117] "RemoveContainer" containerID="1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba" Jul 24 22:39:39.719940 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.719703 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720007 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.719927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720007 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.719978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720099 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720099 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home" (OuterVolumeSpecName: "home") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:39:39.720099 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720229 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720097 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96k9p\" (UniqueName: \"kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p\") pod \"f644919c-eacb-467b-9771-ea6c51bbd43b\" (UID: \"f644919c-eacb-467b-9771-ea6c51bbd43b\") " Jul 24 22:39:39.720229 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:39:39.720319 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:39:39.720475 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720452 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:39.720475 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720473 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:39.720681 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.720484 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:39.722677 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.722648 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p" (OuterVolumeSpecName: "kube-api-access-96k9p") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "kube-api-access-96k9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:39:39.722779 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.722672 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:39:39.723146 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.723106 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm" (OuterVolumeSpecName: "dshm") pod "f644919c-eacb-467b-9771-ea6c51bbd43b" (UID: "f644919c-eacb-467b-9771-ea6c51bbd43b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:39:39.739906 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.739892 2576 scope.go:117] "RemoveContainer" containerID="1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba" Jul 24 22:39:39.740123 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:39:39.740107 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba\": container with ID starting with 1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba not found: ID does not exist" containerID="1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba" Jul 24 22:39:39.740161 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.740130 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba"} err="failed to get container status \"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba\": rpc error: code = NotFound desc = could not find container \"1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba\": container with ID starting with 1b0d476ffb0ad97deb2eed07f142b193e3d59df4a72028e8bca5a662676d2bba not found: ID does not exist" Jul 24 22:39:39.821515 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.821495 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f644919c-eacb-467b-9771-ea6c51bbd43b-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:39.821515 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.821514 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f644919c-eacb-467b-9771-ea6c51bbd43b-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:39.821636 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:39.821524 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96k9p\" (UniqueName: \"kubernetes.io/projected/f644919c-eacb-467b-9771-ea6c51bbd43b-kube-api-access-96k9p\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:39:40.026248 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:40.026188 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:39:40.030435 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:40.030414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-94df6b07-kserve-7df9fc9f67-pf9zk"] Jul 24 22:39:41.890422 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:41.890389 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f644919c-eacb-467b-9771-ea6c51bbd43b" path="/var/lib/kubelet/pods/f644919c-eacb-467b-9771-ea6c51bbd43b/volumes" Jul 24 22:39:43.350571 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:43.350515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:39:53.362230 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:39:53.362193 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:05.762190 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:05.762156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:16.746166 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:16.746136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:25.955524 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:25.955493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:36.361056 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:36.361025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:45.459214 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:45.459182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:40:55.256353 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:40:55.256313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:41:08.548146 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:41:08.548118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:41:17.858212 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:41:17.858176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:41:28.153319 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:41:28.153284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:41:40.555040 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:41:40.555010 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:41:52.344945 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:41:52.344911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:42:00.055167 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:42:00.055137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:42:12.749632 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:42:12.749600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:42:24.957403 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:42:24.957364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:42:38.640376 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:42:38.640344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:42:50.862027 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:42:50.861996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:43:12.354897 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:43:12.354824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:43:23.153198 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:43:23.153168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:43:58.850511 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:43:58.850480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:44:11.354672 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:44:11.354639 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:44:42.744734 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:44:42.744698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:44:54.952920 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:44:54.952887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:45:26.648032 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:45:26.648002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:45:39.549139 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:45:39.549105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:46:10.848583 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:46:10.848542 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:47:38.256918 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:47:38.256884 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:47:48.050025 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:47:48.049994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:47:57.056527 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:47:57.056492 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:48:16.552396 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:16.552367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:48:51.753915 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:51.753882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:48:59.799887 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.799851 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:48:59.800380 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.800228 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f644919c-eacb-467b-9771-ea6c51bbd43b" containerName="main" Jul 24 22:48:59.800380 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.800239 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f644919c-eacb-467b-9771-ea6c51bbd43b" containerName="main" Jul 24 22:48:59.800380 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.800302 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f644919c-eacb-467b-9771-ea6c51bbd43b" containerName="main" Jul 24 22:48:59.803586 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.803567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.806239 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.806215 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-540d8b1d\"/\"kube-root-ca.crt\"" Jul 24 22:48:59.806373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.806271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-540d8b1d\"/\"e2e-lifecycle-model-540d8b1d-kserve-self-signed-certs\"" Jul 24 22:48:59.806373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.806227 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-540d8b1d\"/\"default-dockercfg-tcp7x\"" Jul 24 22:48:59.806373 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.806293 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-540d8b1d\"/\"openshift-service-ca.crt\"" Jul 24 22:48:59.813713 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.813689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:48:59.921417 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.921574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.921574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.921574 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8sv\" (UniqueName: \"kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.921686 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:48:59.921686 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:48:59.921620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.022947 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.022918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.022956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8sv\" (UniqueName: \"kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.022985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023459 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.023630 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.023492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.025311 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.025296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.025498 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.025480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.030089 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.030070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8sv\" (UniqueName: \"kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv\") pod \"e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.113514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.113481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:00.242832 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:00.242807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:49:00.244755 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:49:00.244733 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod745f2eae_aea3_4b29_b5af_21a7a0e38983.slice/crio-a1dce3615ed2cacef25c6df4e6fe6bf155616800c418fdc62591a9b6828082c4 WatchSource:0}: Error finding container a1dce3615ed2cacef25c6df4e6fe6bf155616800c418fdc62591a9b6828082c4: Status 404 returned error can't find the container with id a1dce3615ed2cacef25c6df4e6fe6bf155616800c418fdc62591a9b6828082c4 Jul 24 22:49:01.250231 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:01.250198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" event={"ID":"745f2eae-aea3-4b29-b5af-21a7a0e38983","Type":"ContainerStarted","Data":"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c"} Jul 24 22:49:01.250612 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:01.250237 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" event={"ID":"745f2eae-aea3-4b29-b5af-21a7a0e38983","Type":"ContainerStarted","Data":"a1dce3615ed2cacef25c6df4e6fe6bf155616800c418fdc62591a9b6828082c4"} Jul 24 22:49:03.247309 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:03.247229 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" podStartSLOduration=4.247212493 podStartE2EDuration="4.247212493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:49:01.265224353 +0000 UTC m=+2063.932235173" watchObservedRunningTime="2026-07-24 22:49:03.247212493 +0000 UTC m=+2065.914223314" Jul 24 22:49:03.247757 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:03.247741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:49:10.113847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:10.113816 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:10.113847 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:10.113857 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:10.126076 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:10.126055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:10.299853 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:10.299824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:49:29.554739 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:29.554699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:49:50.548715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:50.548679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:49:58.359159 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:49:58.359125 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:50:04.231185 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:04.231145 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:50:19.251347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.251302 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" podUID="745f2eae-aea3-4b29-b5af-21a7a0e38983" containerName="main" containerID="cri-o://128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c" gracePeriod=45 Jul 24 22:50:19.505101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.505036 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:50:19.595301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595301 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595337 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq8sv\" (UniqueName: \"kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595407 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595610 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm\") pod \"745f2eae-aea3-4b29-b5af-21a7a0e38983\" (UID: \"745f2eae-aea3-4b29-b5af-21a7a0e38983\") " Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595650 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595667 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home" (OuterVolumeSpecName: "home") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache" (OuterVolumeSpecName: "model-cache") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595770 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595806 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.595826 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.595815 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.597778 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.597752 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm" (OuterVolumeSpecName: "dshm") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:50:19.597850 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.597778 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv" (OuterVolumeSpecName: "kube-api-access-bq8sv") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "kube-api-access-bq8sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:50:19.597886 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.597846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "745f2eae-aea3-4b29-b5af-21a7a0e38983" (UID: "745f2eae-aea3-4b29-b5af-21a7a0e38983"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:50:19.611714 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.611684 2576 generic.go:356] "Generic (PLEG): container finished" podID="745f2eae-aea3-4b29-b5af-21a7a0e38983" containerID="128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c" exitCode=0 Jul 24 22:50:19.611837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.611735 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" Jul 24 22:50:19.611837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.611762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" event={"ID":"745f2eae-aea3-4b29-b5af-21a7a0e38983","Type":"ContainerDied","Data":"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c"} Jul 24 22:50:19.611837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.611804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg" event={"ID":"745f2eae-aea3-4b29-b5af-21a7a0e38983","Type":"ContainerDied","Data":"a1dce3615ed2cacef25c6df4e6fe6bf155616800c418fdc62591a9b6828082c4"} Jul 24 22:50:19.611837 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.611820 2576 scope.go:117] "RemoveContainer" containerID="128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c" Jul 24 22:50:19.620985 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.620967 2576 scope.go:117] "RemoveContainer" containerID="128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c" Jul 24 22:50:19.621216 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:50:19.621200 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c\": container with ID starting with 128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c not found: ID does not exist" containerID="128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c" Jul 24 22:50:19.621259 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.621223 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c"} err="failed to get container status \"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c\": rpc error: code = NotFound desc = could not find container \"128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c\": container with ID starting with 128feb4b056fcd28afb48653769f3bfdffb4b31e8c17720cb77b5d851728df0c not found: ID does not exist" Jul 24 22:50:19.631052 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.631020 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:50:19.637009 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.636984 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-540d8b1d/e2e-lifecycle-model-540d8b1d-kserve-65867b7f5f-6w2fg"] Jul 24 22:50:19.696885 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.696850 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745f2eae-aea3-4b29-b5af-21a7a0e38983-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.696885 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.696880 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8sv\" (UniqueName: \"kubernetes.io/projected/745f2eae-aea3-4b29-b5af-21a7a0e38983-kube-api-access-bq8sv\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.696885 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.696889 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/745f2eae-aea3-4b29-b5af-21a7a0e38983-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:50:19.890957 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:50:19.890912 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745f2eae-aea3-4b29-b5af-21a7a0e38983" path="/var/lib/kubelet/pods/745f2eae-aea3-4b29-b5af-21a7a0e38983/volumes" Jul 24 22:51:04.958822 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:51:04.958784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:51:17.706299 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:51:17.706261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:52:03.651583 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:03.651520 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:52:17.650540 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:17.650500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:52:23.514048 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.514017 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:52:23.514409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.514360 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745f2eae-aea3-4b29-b5af-21a7a0e38983" containerName="main" Jul 24 22:52:23.514409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.514370 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="745f2eae-aea3-4b29-b5af-21a7a0e38983" containerName="main" Jul 24 22:52:23.514476 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.514429 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="745f2eae-aea3-4b29-b5af-21a7a0e38983" containerName="main" Jul 24 22:52:23.517017 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.517001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.519765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.519736 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11557e\"/\"default-dockercfg-hhfwr\"" Jul 24 22:52:23.519765 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.519743 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11557e\"/\"openshift-service-ca.crt\"" Jul 24 22:52:23.519955 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.519857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11557e\"/\"test-model-11557e-kserve-self-signed-certs\"" Jul 24 22:52:23.520021 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.520001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11557e\"/\"kube-root-ca.crt\"" Jul 24 22:52:23.525685 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.525664 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:52:23.628227 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.628227 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.628402 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2pb\" (UniqueName: \"kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.628402 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.628468 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.628468 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.628451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729451 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2pb\" (UniqueName: \"kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729451 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729667 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729891 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.729948 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.730008 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.729990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.732074 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.732046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.732264 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.732249 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.737298 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.737277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2pb\" (UniqueName: \"kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb\") pod \"test-model-11557e-kserve-6cd8cd5669-7tw92\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.826975 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.826912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:23.963063 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:23.962856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:52:23.965444 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:52:23.965421 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e7191e_ff79_4443_b38a_de064a48881a.slice/crio-263327f45b335f2f1af4f4ee5abcee16715ef1a2faf450fde3fd5723860229aa WatchSource:0}: Error finding container 263327f45b335f2f1af4f4ee5abcee16715ef1a2faf450fde3fd5723860229aa: Status 404 returned error can't find the container with id 263327f45b335f2f1af4f4ee5abcee16715ef1a2faf450fde3fd5723860229aa Jul 24 22:52:24.190606 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:24.190570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" event={"ID":"50e7191e-ff79-4443-b38a-de064a48881a","Type":"ContainerStarted","Data":"bbc7e7842d00ebf29bf094c715d633511caf5a5148b984d9f23caee5746fffb3"} Jul 24 22:52:24.190781 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:24.190617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" event={"ID":"50e7191e-ff79-4443-b38a-de064a48881a","Type":"ContainerStarted","Data":"263327f45b335f2f1af4f4ee5abcee16715ef1a2faf450fde3fd5723860229aa"} Jul 24 22:52:24.211828 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:24.211776 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" podStartSLOduration=1.211760453 podStartE2EDuration="1.211760453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:24.210273916 +0000 UTC m=+2266.877284736" watchObservedRunningTime="2026-07-24 22:52:24.211760453 +0000 UTC m=+2266.878771266" Jul 24 22:52:28.866280 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:28.866243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:52:33.827889 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:33.827854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:33.828365 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:33.827900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:33.840106 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:33.840082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:34.251626 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:34.251596 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:52:40.449815 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:52:40.449782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:53:00.106190 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.106145 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:00.110392 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.110355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.113575 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.113367 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-422c4e\"/\"kube-root-ca.crt\"" Jul 24 22:53:00.113575 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.113368 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-422c4e\"/\"test-model-422c4e-kserve-self-signed-certs\"" Jul 24 22:53:00.113575 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.113367 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-422c4e\"/\"openshift-service-ca.crt\"" Jul 24 22:53:00.113575 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.113452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-422c4e\"/\"default-dockercfg-p2dw2\"" Jul 24 22:53:00.122289 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.122267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:00.259643 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.259807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.259807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7vb\" (UniqueName: \"kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.259807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.259807 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.259960 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.259866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361382 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361382 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361678 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7vb\" (UniqueName: \"kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361678 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361678 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361678 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361917 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361917 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361859 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.361917 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.361875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.363693 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.363673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.364043 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.364023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.369131 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.369108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7vb\" (UniqueName: \"kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb\") pod \"test-model-422c4e-kserve-774d967fdf-tvq54\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.421740 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.421700 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:00.556179 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:00.556152 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:00.559038 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:53:00.559004 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfedbcf0c_b6d9_4598_bc62_1509faa20eda.slice/crio-eb5cd950d66d9828d5aa3208c6c27dd36447d00839abb79bddf927d381f45d7b WatchSource:0}: Error finding container eb5cd950d66d9828d5aa3208c6c27dd36447d00839abb79bddf927d381f45d7b: Status 404 returned error can't find the container with id eb5cd950d66d9828d5aa3208c6c27dd36447d00839abb79bddf927d381f45d7b Jul 24 22:53:01.362862 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:01.362823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" event={"ID":"fedbcf0c-b6d9-4598-bc62-1509faa20eda","Type":"ContainerStarted","Data":"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059"} Jul 24 22:53:01.363278 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:01.362872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" event={"ID":"fedbcf0c-b6d9-4598-bc62-1509faa20eda","Type":"ContainerStarted","Data":"eb5cd950d66d9828d5aa3208c6c27dd36447d00839abb79bddf927d381f45d7b"} Jul 24 22:53:01.379633 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:01.379563 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" podStartSLOduration=1.379534622 podStartE2EDuration="1.379534622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:01.379098469 +0000 UTC m=+2304.046109291" watchObservedRunningTime="2026-07-24 22:53:01.379534622 +0000 UTC m=+2304.046545434" Jul 24 22:53:04.549176 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:04.549137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:53:10.422243 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:10.422210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:10.422647 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:10.422267 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:10.434819 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:10.434798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:11.421201 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:11.421171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:17.945419 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:17.945386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:53:41.412422 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:41.412390 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:53:42.332759 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:42.332723 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:45.254987 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:45.254956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:53:56.432341 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.432296 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" podUID="50e7191e-ff79-4443-b38a-de064a48881a" containerName="main" containerID="cri-o://bbc7e7842d00ebf29bf094c715d633511caf5a5148b984d9f23caee5746fffb3" gracePeriod=45 Jul 24 22:53:56.626860 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.626809 2576 generic.go:356] "Generic (PLEG): container finished" podID="50e7191e-ff79-4443-b38a-de064a48881a" containerID="bbc7e7842d00ebf29bf094c715d633511caf5a5148b984d9f23caee5746fffb3" exitCode=0 Jul 24 22:53:56.627019 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.626885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" event={"ID":"50e7191e-ff79-4443-b38a-de064a48881a","Type":"ContainerDied","Data":"bbc7e7842d00ebf29bf094c715d633511caf5a5148b984d9f23caee5746fffb3"} Jul 24 22:53:56.684005 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.683952 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:53:56.763119 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763249 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763133 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r2pb\" (UniqueName: \"kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763249 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763249 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763175 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763249 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763249 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs\") pod \"50e7191e-ff79-4443-b38a-de064a48881a\" (UID: \"50e7191e-ff79-4443-b38a-de064a48881a\") " Jul 24 22:53:56.763518 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763328 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache" (OuterVolumeSpecName: "model-cache") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:56.763518 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763394 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:56.763518 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763416 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home" (OuterVolumeSpecName: "home") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:56.763685 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763619 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:56.763685 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763643 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:56.763685 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.763658 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:56.765715 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.765688 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:53:56.765791 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.765737 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb" (OuterVolumeSpecName: "kube-api-access-8r2pb") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "kube-api-access-8r2pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:53:56.765791 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.765760 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm" (OuterVolumeSpecName: "dshm") pod "50e7191e-ff79-4443-b38a-de064a48881a" (UID: "50e7191e-ff79-4443-b38a-de064a48881a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:56.864354 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.864329 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2pb\" (UniqueName: \"kubernetes.io/projected/50e7191e-ff79-4443-b38a-de064a48881a-kube-api-access-8r2pb\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:56.864354 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.864350 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e7191e-ff79-4443-b38a-de064a48881a-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:56.864486 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:56.864360 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e7191e-ff79-4443-b38a-de064a48881a-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.352395 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.352361 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" podUID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" containerName="main" containerID="cri-o://ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059" gracePeriod=45 Jul 24 22:53:57.612663 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.612606 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:57.634932 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.634901 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" Jul 24 22:53:57.634932 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.634910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92" event={"ID":"50e7191e-ff79-4443-b38a-de064a48881a","Type":"ContainerDied","Data":"263327f45b335f2f1af4f4ee5abcee16715ef1a2faf450fde3fd5723860229aa"} Jul 24 22:53:57.635131 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.634955 2576 scope.go:117] "RemoveContainer" containerID="bbc7e7842d00ebf29bf094c715d633511caf5a5148b984d9f23caee5746fffb3" Jul 24 22:53:57.636325 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.636299 2576 generic.go:356] "Generic (PLEG): container finished" podID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" containerID="ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059" exitCode=0 Jul 24 22:53:57.636465 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.636347 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" Jul 24 22:53:57.636465 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.636345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" event={"ID":"fedbcf0c-b6d9-4598-bc62-1509faa20eda","Type":"ContainerDied","Data":"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059"} Jul 24 22:53:57.636465 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.636389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54" event={"ID":"fedbcf0c-b6d9-4598-bc62-1509faa20eda","Type":"ContainerDied","Data":"eb5cd950d66d9828d5aa3208c6c27dd36447d00839abb79bddf927d381f45d7b"} Jul 24 22:53:57.644163 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.644146 2576 scope.go:117] "RemoveContainer" containerID="ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059" Jul 24 22:53:57.652811 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.652795 2576 scope.go:117] "RemoveContainer" containerID="ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059" Jul 24 22:53:57.653052 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:53:57.653035 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059\": container with ID starting with ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059 not found: ID does not exist" containerID="ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059" Jul 24 22:53:57.653116 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.653064 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059"} err="failed to get container status \"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059\": rpc error: code = NotFound desc = could not find container \"ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059\": container with ID starting with ff4bfa59cb46c812eae289968a727ca6b19b9a35a737a5f26a2bdb8db28b8059 not found: ID does not exist" Jul 24 22:53:57.655777 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.655757 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:53:57.659882 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.659865 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-11557e/test-model-11557e-kserve-6cd8cd5669-7tw92"] Jul 24 22:53:57.670392 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670479 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670479 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc7vb\" (UniqueName: \"kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670564 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670489 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670564 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670669 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670580 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs\") pod \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\" (UID: \"fedbcf0c-b6d9-4598-bc62-1509faa20eda\") " Jul 24 22:53:57.670817 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670768 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache" (OuterVolumeSpecName: "model-cache") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:57.670817 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670790 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home" (OuterVolumeSpecName: "home") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:57.670817 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670802 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:57.671016 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670889 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.671016 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670900 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.671016 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.670908 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.672674 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.672644 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm" (OuterVolumeSpecName: "dshm") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:53:57.672758 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.672705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:53:57.672758 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.672714 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb" (OuterVolumeSpecName: "kube-api-access-zc7vb") pod "fedbcf0c-b6d9-4598-bc62-1509faa20eda" (UID: "fedbcf0c-b6d9-4598-bc62-1509faa20eda"). InnerVolumeSpecName "kube-api-access-zc7vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:53:57.771783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.771764 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedbcf0c-b6d9-4598-bc62-1509faa20eda-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.771783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.771783 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedbcf0c-b6d9-4598-bc62-1509faa20eda-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.771903 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.771792 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7vb\" (UniqueName: \"kubernetes.io/projected/fedbcf0c-b6d9-4598-bc62-1509faa20eda-kube-api-access-zc7vb\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:53:57.893835 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.893761 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e7191e-ff79-4443-b38a-de064a48881a" path="/var/lib/kubelet/pods/50e7191e-ff79-4443-b38a-de064a48881a/volumes" Jul 24 22:53:57.950604 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.950573 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:57.958128 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:57.958108 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-422c4e/test-model-422c4e-kserve-774d967fdf-tvq54"] Jul 24 22:53:58.345541 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:58.345439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:53:59.890702 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:53:59.890663 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" path="/var/lib/kubelet/pods/fedbcf0c-b6d9-4598-bc62-1509faa20eda/volumes" Jul 24 22:54:17.554225 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:54:17.554189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:54:37.759462 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:54:37.759426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:54:53.461473 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:54:53.461428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:55:09.457291 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:55:09.457253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:55:38.154878 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:55:38.154839 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:55:53.058728 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:55:53.058694 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:56:05.260429 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:05.260395 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:56:08.002906 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.002870 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003226 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" containerName="main" Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003237 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" containerName="main" Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003249 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e7191e-ff79-4443-b38a-de064a48881a" containerName="main" Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e7191e-ff79-4443-b38a-de064a48881a" containerName="main" Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003318 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedbcf0c-b6d9-4598-bc62-1509faa20eda" containerName="main" Jul 24 22:56:08.003370 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.003331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e7191e-ff79-4443-b38a-de064a48881a" containerName="main" Jul 24 22:56:08.006345 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.006324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.008528 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.008507 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"openshift-service-ca.crt\"" Jul 24 22:56:08.009151 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.009030 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"kube-root-ca.crt\"" Jul 24 22:56:08.009151 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.009042 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"default-dockercfg-rkxt2\"" Jul 24 22:56:08.009151 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.009072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"auth-test-model-499dd1-kserve-self-signed-certs\"" Jul 24 22:56:08.014611 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.014585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:56:08.091577 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.091726 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.091726 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.091726 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.091726 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvc9\" (UniqueName: \"kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.091863 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.091764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192800 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvc9\" (UniqueName: \"kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.192990 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.192963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.193339 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.193314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.193408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.193349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.193408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.193396 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.195397 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.195378 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.195456 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.195441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.201381 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.201356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvc9\" (UniqueName: \"kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9\") pod \"auth-test-model-499dd1-kserve-54b4869567-p8qxw\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.318481 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.318401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:08.447886 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:08.447861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:56:08.450524 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:56:08.450497 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89e2015d_45c9_4a03_ac1f_21a14b0e3b43.slice/crio-f397b295afa4e97c673b96785e0d6521d6cb18dd3785026708b35f04083ba497 WatchSource:0}: Error finding container f397b295afa4e97c673b96785e0d6521d6cb18dd3785026708b35f04083ba497: Status 404 returned error can't find the container with id f397b295afa4e97c673b96785e0d6521d6cb18dd3785026708b35f04083ba497 Jul 24 22:56:09.245385 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:09.245352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" event={"ID":"89e2015d-45c9-4a03-ac1f-21a14b0e3b43","Type":"ContainerStarted","Data":"2225df36bf93e52230838b06755af0b9887b7ee9fdef5c99a66c8f42f9c06f1f"} Jul 24 22:56:09.245911 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:09.245396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" event={"ID":"89e2015d-45c9-4a03-ac1f-21a14b0e3b43","Type":"ContainerStarted","Data":"f397b295afa4e97c673b96785e0d6521d6cb18dd3785026708b35f04083ba497"} Jul 24 22:56:09.266536 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:09.266474 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" podStartSLOduration=2.266454829 podStartE2EDuration="2.266454829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:09.265683857 +0000 UTC m=+2491.932694678" watchObservedRunningTime="2026-07-24 22:56:09.266454829 +0000 UTC m=+2491.933465629" Jul 24 22:56:13.445850 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:13.445815 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:56:18.318751 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:18.318719 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:18.321086 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:18.318784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:18.331029 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:18.331008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:19.302347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:19.302318 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:56:33.204714 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.204682 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:56:33.208101 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.208081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.210831 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.210806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"default-dockercfg-5pnjw\"" Jul 24 22:56:33.210950 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.210838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"openshift-service-ca.crt\"" Jul 24 22:56:33.210950 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.210806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"auth-test-model-e15986-kserve-self-signed-certs\"" Jul 24 22:56:33.210950 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.210909 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"kube-root-ca.crt\"" Jul 24 22:56:33.219197 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.219174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:56:33.317687 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.317856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mwl\" (UniqueName: \"kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.317856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.317856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.317856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.317856 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.317852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418291 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418291 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418514 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mwl\" (UniqueName: \"kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418734 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418798 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.418853 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.418795 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.420869 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.420845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.421095 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.421079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.425275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.425250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mwl\" (UniqueName: \"kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl\") pod \"auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.518646 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.518535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:33.652276 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:33.652251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:56:33.654704 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:56:33.654674 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc0c370_900c_4085_970e_6c0f284dcabf.slice/crio-23127cca243342f01aa3bb23bbdc5b86858bea21bf2696a1737cbd882765582f WatchSource:0}: Error finding container 23127cca243342f01aa3bb23bbdc5b86858bea21bf2696a1737cbd882765582f: Status 404 returned error can't find the container with id 23127cca243342f01aa3bb23bbdc5b86858bea21bf2696a1737cbd882765582f Jul 24 22:56:34.362244 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:34.362204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" event={"ID":"0fc0c370-900c-4085-970e-6c0f284dcabf","Type":"ContainerStarted","Data":"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26"} Jul 24 22:56:34.362244 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:34.362248 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" event={"ID":"0fc0c370-900c-4085-970e-6c0f284dcabf","Type":"ContainerStarted","Data":"23127cca243342f01aa3bb23bbdc5b86858bea21bf2696a1737cbd882765582f"} Jul 24 22:56:34.378289 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:34.378245 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" podStartSLOduration=1.378229475 podStartE2EDuration="1.378229475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:34.376441227 +0000 UTC m=+2517.043452050" watchObservedRunningTime="2026-07-24 22:56:34.378229475 +0000 UTC m=+2517.045240296" Jul 24 22:56:36.559880 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:36.559845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:56:43.519743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:43.519709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:43.520275 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:43.519766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:43.532274 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:43.532248 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:44.427668 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:44.427640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:56:53.948166 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:56:53.948129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:57:11.953189 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:57:11.953153 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:57:28.946408 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:57:28.946371 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:57:46.943783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:57:46.943733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:58:05.247891 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:05.247797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:58:22.043753 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:22.043721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:58:32.849800 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:32.849768 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:58:33.690409 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:33.690379 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:58:39.849178 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:39.849135 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:58:45.707340 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.707309 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 22:58:45.710900 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.710884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.712908 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.712882 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"sub-test-model-499dd1-kserve-self-signed-certs\"" Jul 24 22:58:45.721123 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.721103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 22:58:45.790350 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8k6j\" (UniqueName: \"kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.790461 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.790507 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.790507 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.790655 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.790655 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.790610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891446 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891578 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891578 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891578 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891707 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8k6j\" (UniqueName: \"kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891707 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891872 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891960 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.891999 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.891982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.893789 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.893772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.894096 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.894080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:45.899484 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:45.899465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8k6j\" (UniqueName: \"kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j\") pod \"sub-test-model-499dd1-kserve-7f497578d5-5kqmg\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:46.019743 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:46.019681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:46.153697 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:46.152596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 22:58:46.988704 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:46.988670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" event={"ID":"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841","Type":"ContainerStarted","Data":"46982679bea70ef8f736bb94a576d5a3ec0b66197a9dcd5036f73134333163a2"} Jul 24 22:58:46.989061 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:46.988720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" event={"ID":"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841","Type":"ContainerStarted","Data":"762b3646bf2ddd75c06b15ca1363aaf27c8b542276c5af183a2f64c6f9daf024"} Jul 24 22:58:47.003624 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:47.003570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" podStartSLOduration=2.003535583 podStartE2EDuration="2.003535583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:58:47.002904309 +0000 UTC m=+2649.669915130" watchObservedRunningTime="2026-07-24 22:58:47.003535583 +0000 UTC m=+2649.670546402" Jul 24 22:58:47.869176 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:47.869117 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" podUID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" containerName="main" containerID="cri-o://2225df36bf93e52230838b06755af0b9887b7ee9fdef5c99a66c8f42f9c06f1f" gracePeriod=45 Jul 24 22:58:47.995460 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:47.995432 2576 generic.go:356] "Generic (PLEG): container finished" podID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" containerID="2225df36bf93e52230838b06755af0b9887b7ee9fdef5c99a66c8f42f9c06f1f" exitCode=0 Jul 24 22:58:47.995896 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:47.995500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" event={"ID":"89e2015d-45c9-4a03-ac1f-21a14b0e3b43","Type":"ContainerDied","Data":"2225df36bf93e52230838b06755af0b9887b7ee9fdef5c99a66c8f42f9c06f1f"} Jul 24 22:58:48.118668 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.118643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:58:48.211973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.211891 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.211973 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.211927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.212193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.211984 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.212193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.212193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212063 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvvc9\" (UniqueName: \"kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.212193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm\") pod \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\" (UID: \"89e2015d-45c9-4a03-ac1f-21a14b0e3b43\") " Jul 24 22:58:48.212193 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212130 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home" (OuterVolumeSpecName: "home") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:48.212413 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:48.212413 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212352 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache" (OuterVolumeSpecName: "model-cache") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:48.212413 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.212362 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.214351 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.214332 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:58:48.214428 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.214371 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm" (OuterVolumeSpecName: "dshm") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:48.214466 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.214432 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9" (OuterVolumeSpecName: "kube-api-access-pvvc9") pod "89e2015d-45c9-4a03-ac1f-21a14b0e3b43" (UID: "89e2015d-45c9-4a03-ac1f-21a14b0e3b43"). InnerVolumeSpecName "kube-api-access-pvvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:58:48.312967 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.312934 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.312967 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.312967 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.313165 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.312979 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvc9\" (UniqueName: \"kubernetes.io/projected/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-kube-api-access-pvvc9\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.313165 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.312992 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.313165 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.313004 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e2015d-45c9-4a03-ac1f-21a14b0e3b43-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:48.711061 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.711017 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" podUID="0fc0c370-900c-4085-970e-6c0f284dcabf" containerName="main" containerID="cri-o://b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26" gracePeriod=45 Jul 24 22:58:48.949873 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:48.949850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:58:49.001750 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.001679 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" Jul 24 22:58:49.001750 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.001723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw" event={"ID":"89e2015d-45c9-4a03-ac1f-21a14b0e3b43","Type":"ContainerDied","Data":"f397b295afa4e97c673b96785e0d6521d6cb18dd3785026708b35f04083ba497"} Jul 24 22:58:49.002212 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.001775 2576 scope.go:117] "RemoveContainer" containerID="2225df36bf93e52230838b06755af0b9887b7ee9fdef5c99a66c8f42f9c06f1f" Jul 24 22:58:49.003119 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.003098 2576 generic.go:356] "Generic (PLEG): container finished" podID="0fc0c370-900c-4085-970e-6c0f284dcabf" containerID="b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26" exitCode=0 Jul 24 22:58:49.003208 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.003137 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" Jul 24 22:58:49.003208 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.003162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" event={"ID":"0fc0c370-900c-4085-970e-6c0f284dcabf","Type":"ContainerDied","Data":"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26"} Jul 24 22:58:49.003208 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.003190 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5" event={"ID":"0fc0c370-900c-4085-970e-6c0f284dcabf","Type":"ContainerDied","Data":"23127cca243342f01aa3bb23bbdc5b86858bea21bf2696a1737cbd882765582f"} Jul 24 22:58:49.011310 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.011291 2576 scope.go:117] "RemoveContainer" containerID="b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26" Jul 24 22:58:49.018876 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.018829 2576 scope.go:117] "RemoveContainer" containerID="b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26" Jul 24 22:58:49.019127 ip-10-0-132-132 kubenswrapper[2576]: E0724 22:58:49.019104 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26\": container with ID starting with b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26 not found: ID does not exist" containerID="b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26" Jul 24 22:58:49.019212 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019127 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019212 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019195 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mwl\" (UniqueName: \"kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019347 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019442 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019357 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019516 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:49.019612 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019138 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26"} err="failed to get container status \"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26\": rpc error: code = NotFound desc = could not find container \"b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26\": container with ID starting with b557414a15a487d37d52fa6181a7b3b30f3f6d20d858b559e808ef24aa44bb26 not found: ID does not exist" Jul 24 22:58:49.019797 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm\") pod \"0fc0c370-900c-4085-970e-6c0f284dcabf\" (UID: \"0fc0c370-900c-4085-970e-6c0f284dcabf\") " Jul 24 22:58:49.019859 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019788 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:49.019859 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.019800 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home" (OuterVolumeSpecName: "home") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:49.020091 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.020071 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.020091 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.020091 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.020228 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.020103 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.021391 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.021368 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:58:49.021691 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.021672 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl" (OuterVolumeSpecName: "kube-api-access-t5mwl") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "kube-api-access-t5mwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:58:49.021757 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.021696 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:58:49.021757 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.021742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm" (OuterVolumeSpecName: "dshm") pod "0fc0c370-900c-4085-970e-6c0f284dcabf" (UID: "0fc0c370-900c-4085-970e-6c0f284dcabf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:58:49.026112 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.026092 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/auth-test-model-499dd1-kserve-54b4869567-p8qxw"] Jul 24 22:58:49.121184 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.121160 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc0c370-900c-4085-970e-6c0f284dcabf-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.121265 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.121185 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mwl\" (UniqueName: \"kubernetes.io/projected/0fc0c370-900c-4085-970e-6c0f284dcabf-kube-api-access-t5mwl\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.121265 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.121198 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc0c370-900c-4085-970e-6c0f284dcabf-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 22:58:49.322315 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.322294 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:58:49.326446 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.326424 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/auth-test-model-e15986-kserve-78cd8cc5bc-4ndc5"] Jul 24 22:58:49.889532 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.889484 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc0c370-900c-4085-970e-6c0f284dcabf" path="/var/lib/kubelet/pods/0fc0c370-900c-4085-970e-6c0f284dcabf/volumes" Jul 24 22:58:49.889907 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:49.889892 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" path="/var/lib/kubelet/pods/89e2015d-45c9-4a03-ac1f-21a14b0e3b43/volumes" Jul 24 22:58:50.444830 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:50.444795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:58:56.020006 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:56.019974 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:56.020522 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:56.020021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:56.032109 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:56.032078 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:58:56.052641 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:58:56.052616 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 22:59:40.216285 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216248 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 22:59:40.216695 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216635 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc0c370-900c-4085-970e-6c0f284dcabf" containerName="main" Jul 24 22:59:40.216695 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216647 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc0c370-900c-4085-970e-6c0f284dcabf" containerName="main" Jul 24 22:59:40.216695 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216670 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" containerName="main" Jul 24 22:59:40.216695 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216675 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" containerName="main" Jul 24 22:59:40.216829 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216729 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc0c370-900c-4085-970e-6c0f284dcabf" containerName="main" Jul 24 22:59:40.216829 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.216742 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e2015d-45c9-4a03-ac1f-21a14b0e3b43" containerName="main" Jul 24 22:59:40.221091 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.221075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.223451 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.223434 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"kube-root-ca.crt\"" Jul 24 22:59:40.223939 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.223919 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"openshift-service-ca.crt\"" Jul 24 22:59:40.224012 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.223990 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"sub-test-model-e15986-kserve-self-signed-certs\"" Jul 24 22:59:40.224068 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.223996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"default-dockercfg-5pnjw\"" Jul 24 22:59:40.230578 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.230526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 22:59:40.274117 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.274272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rdn\" (UniqueName: \"kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.274272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.274272 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.274454 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.274454 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.274386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375548 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375548 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rdn\" (UniqueName: \"kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375783 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375874 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375951 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.375997 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.376038 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.375998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.376217 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.376196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.378004 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.377986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.378286 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.378267 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.383332 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.383314 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rdn\" (UniqueName: \"kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn\") pod \"sub-test-model-e15986-kserve-75f8ff448c-lwlpk\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.531541 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.531454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:40.662398 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:40.662367 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 22:59:40.665572 ip-10-0-132-132 kubenswrapper[2576]: W0724 22:59:40.665531 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d9259ce_b7b5_4705_b849_c39aa8483289.slice/crio-ef77f67ad11758a44795fa06e1013772ed4480c25fd80f0ed489fffd5dedcd05 WatchSource:0}: Error finding container ef77f67ad11758a44795fa06e1013772ed4480c25fd80f0ed489fffd5dedcd05: Status 404 returned error can't find the container with id ef77f67ad11758a44795fa06e1013772ed4480c25fd80f0ed489fffd5dedcd05 Jul 24 22:59:41.246519 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:41.246483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" event={"ID":"3d9259ce-b7b5-4705-b849-c39aa8483289","Type":"ContainerStarted","Data":"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4"} Jul 24 22:59:41.246922 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:41.246529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" event={"ID":"3d9259ce-b7b5-4705-b849-c39aa8483289","Type":"ContainerStarted","Data":"ef77f67ad11758a44795fa06e1013772ed4480c25fd80f0ed489fffd5dedcd05"} Jul 24 22:59:41.263336 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:41.263288 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" podStartSLOduration=1.263272664 podStartE2EDuration="1.263272664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:59:41.260791771 +0000 UTC m=+2703.927802595" watchObservedRunningTime="2026-07-24 22:59:41.263272664 +0000 UTC m=+2703.930283485" Jul 24 22:59:43.647972 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:43.647931 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 22:59:50.532497 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:50.532458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:50.532497 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:50.532511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:50.545132 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:50.545108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 22:59:51.304861 ip-10-0-132-132 kubenswrapper[2576]: I0724 22:59:51.304831 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 23:00:05.547370 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:05.547331 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:00:24.949897 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:24.949862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:00:30.399118 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:30.399082 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 23:00:31.189748 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:31.189712 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 23:00:41.555772 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:41.555731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:00:42.506742 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.506699 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:00:42.510749 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.510729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.512572 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.512533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-499dd1\"/\"rate-test-model-499dd1-kserve-self-signed-certs\"" Jul 24 23:00:42.519243 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.519218 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:00:42.650967 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.650932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.650967 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.650969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.651437 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.650996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.651437 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.651108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctd9\" (UniqueName: \"kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.651437 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.651143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.651437 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.651186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752382 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752382 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctd9\" (UniqueName: \"kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752875 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752917 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.752954 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.752923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.754998 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.754975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.755256 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.755237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.772372 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.772298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctd9\" (UniqueName: \"kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9\") pod \"rate-test-model-499dd1-kserve-748899d7b7-52mv9\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.821986 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.821939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:42.960636 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:42.960610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:00:42.963123 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:00:42.963100 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49a519fe_4eeb_498f_8b33_98e7a034f0c2.slice/crio-805549034bd12f1b76d1f88a18c3e069af2a20477d3eaf9b37c63f920164d919 WatchSource:0}: Error finding container 805549034bd12f1b76d1f88a18c3e069af2a20477d3eaf9b37c63f920164d919: Status 404 returned error can't find the container with id 805549034bd12f1b76d1f88a18c3e069af2a20477d3eaf9b37c63f920164d919 Jul 24 23:00:43.547921 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:43.547878 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" event={"ID":"49a519fe-4eeb-498f-8b33-98e7a034f0c2","Type":"ContainerStarted","Data":"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2"} Jul 24 23:00:43.547921 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:43.547926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" event={"ID":"49a519fe-4eeb-498f-8b33-98e7a034f0c2","Type":"ContainerStarted","Data":"805549034bd12f1b76d1f88a18c3e069af2a20477d3eaf9b37c63f920164d919"} Jul 24 23:00:43.563817 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:43.563766 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" podStartSLOduration=1.563748934 podStartE2EDuration="1.563748934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:00:43.562739027 +0000 UTC m=+2766.229749843" watchObservedRunningTime="2026-07-24 23:00:43.563748934 +0000 UTC m=+2766.230759733" Jul 24 23:00:45.418471 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.418429 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" podUID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" containerName="main" containerID="cri-o://46982679bea70ef8f736bb94a576d5a3ec0b66197a9dcd5036f73134333163a2" gracePeriod=45 Jul 24 23:00:45.559066 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.559036 2576 generic.go:356] "Generic (PLEG): container finished" podID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" containerID="46982679bea70ef8f736bb94a576d5a3ec0b66197a9dcd5036f73134333163a2" exitCode=0 Jul 24 23:00:45.559223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.559074 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" event={"ID":"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841","Type":"ContainerDied","Data":"46982679bea70ef8f736bb94a576d5a3ec0b66197a9dcd5036f73134333163a2"} Jul 24 23:00:45.681855 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.681791 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 23:00:45.785808 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.785779 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.785969 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.785821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.785969 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.785911 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.785969 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.785949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.786156 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.785979 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.786156 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786052 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8k6j\" (UniqueName: \"kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j\") pod \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\" (UID: \"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841\") " Jul 24 23:00:45.786266 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:45.786266 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:45.786365 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786327 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home" (OuterVolumeSpecName: "home") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:45.786480 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786463 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:45.786535 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786483 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:45.786535 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.786491 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:45.788146 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.788115 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm" (OuterVolumeSpecName: "dshm") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:45.788146 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.788119 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:00:45.788292 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.788162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j" (OuterVolumeSpecName: "kube-api-access-l8k6j") pod "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" (UID: "d5e60c04-637f-4fc6-ab5f-0bb2b4e41841"). InnerVolumeSpecName "kube-api-access-l8k6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:00:45.887703 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.887676 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:45.887703 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.887699 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:45.887703 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:45.887708 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8k6j\" (UniqueName: \"kubernetes.io/projected/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841-kube-api-access-l8k6j\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.209355 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.209293 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" podUID="3d9259ce-b7b5-4705-b849-c39aa8483289" containerName="main" containerID="cri-o://e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4" gracePeriod=45 Jul 24 23:00:46.470990 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.470930 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 23:00:46.565034 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.565004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" Jul 24 23:00:46.565236 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.565046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg" event={"ID":"d5e60c04-637f-4fc6-ab5f-0bb2b4e41841","Type":"ContainerDied","Data":"762b3646bf2ddd75c06b15ca1363aaf27c8b542276c5af183a2f64c6f9daf024"} Jul 24 23:00:46.565236 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.565093 2576 scope.go:117] "RemoveContainer" containerID="46982679bea70ef8f736bb94a576d5a3ec0b66197a9dcd5036f73134333163a2" Jul 24 23:00:46.566476 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.566446 2576 generic.go:356] "Generic (PLEG): container finished" podID="3d9259ce-b7b5-4705-b849-c39aa8483289" containerID="e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4" exitCode=0 Jul 24 23:00:46.566616 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.566489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" event={"ID":"3d9259ce-b7b5-4705-b849-c39aa8483289","Type":"ContainerDied","Data":"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4"} Jul 24 23:00:46.566616 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.566518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" event={"ID":"3d9259ce-b7b5-4705-b849-c39aa8483289","Type":"ContainerDied","Data":"ef77f67ad11758a44795fa06e1013772ed4480c25fd80f0ed489fffd5dedcd05"} Jul 24 23:00:46.566616 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.566489 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk" Jul 24 23:00:46.575664 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.575636 2576 scope.go:117] "RemoveContainer" containerID="e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4" Jul 24 23:00:46.579621 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.579599 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 23:00:46.583357 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.583335 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/sub-test-model-499dd1-kserve-7f497578d5-5kqmg"] Jul 24 23:00:46.584465 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.584450 2576 scope.go:117] "RemoveContainer" containerID="e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4" Jul 24 23:00:46.584737 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:00:46.584715 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4\": container with ID starting with e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4 not found: ID does not exist" containerID="e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4" Jul 24 23:00:46.584830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.584744 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4"} err="failed to get container status \"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4\": rpc error: code = NotFound desc = could not find container \"e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4\": container with ID starting with e23d506a7816ecdbc58bcde2760c7752819922b38d2e4cd48ac45fb0b7bb78a4 not found: ID does not exist" Jul 24 23:00:46.595973 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.595949 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596072 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7rdn\" (UniqueName: \"kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596116 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596073 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596116 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596233 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596216 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596288 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596244 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:46.596341 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home\") pod \"3d9259ce-b7b5-4705-b849-c39aa8483289\" (UID: \"3d9259ce-b7b5-4705-b849-c39aa8483289\") " Jul 24 23:00:46.596341 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596307 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:46.596590 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home" (OuterVolumeSpecName: "home") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:46.596762 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596641 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.596762 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596658 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.596762 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.596668 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.598348 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.598328 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm" (OuterVolumeSpecName: "dshm") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:00:46.598450 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.598409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn" (OuterVolumeSpecName: "kube-api-access-h7rdn") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "kube-api-access-h7rdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:00:46.598512 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.598495 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d9259ce-b7b5-4705-b849-c39aa8483289" (UID: "3d9259ce-b7b5-4705-b849-c39aa8483289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:00:46.698083 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.698044 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9259ce-b7b5-4705-b849-c39aa8483289-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.698083 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.698077 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rdn\" (UniqueName: \"kubernetes.io/projected/3d9259ce-b7b5-4705-b849-c39aa8483289-kube-api-access-h7rdn\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.698083 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.698087 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9259ce-b7b5-4705-b849-c39aa8483289-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:00:46.887036 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.887008 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 23:00:46.890763 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:46.890740 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/sub-test-model-e15986-kserve-75f8ff448c-lwlpk"] Jul 24 23:00:47.890320 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:47.890285 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9259ce-b7b5-4705-b849-c39aa8483289" path="/var/lib/kubelet/pods/3d9259ce-b7b5-4705-b849-c39aa8483289/volumes" Jul 24 23:00:47.890727 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:47.890713 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" path="/var/lib/kubelet/pods/d5e60c04-637f-4fc6-ab5f-0bb2b4e41841/volumes" Jul 24 23:00:50.850603 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:50.850565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:00:52.822949 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:52.822908 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:52.822949 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:52.822956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:52.835024 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:52.834993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:00:53.611278 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:00:53.611245 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:01:02.400160 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400078 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400434 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9259ce-b7b5-4705-b849-c39aa8483289" containerName="main" Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400444 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9259ce-b7b5-4705-b849-c39aa8483289" containerName="main" Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400464 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" containerName="main" Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400470 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" containerName="main" Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400538 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e60c04-637f-4fc6-ab5f-0bb2b4e41841" containerName="main" Jul 24 23:01:02.400653 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.400545 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9259ce-b7b5-4705-b849-c39aa8483289" containerName="main" Jul 24 23:01:02.405501 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.405482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.407978 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.407957 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"openshift-service-ca.crt\"" Jul 24 23:01:02.407978 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.407966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"rate-test-model-e15986-kserve-self-signed-certs\"" Jul 24 23:01:02.408152 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.407960 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"default-dockercfg-5pnjw\"" Jul 24 23:01:02.408152 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.407957 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e15986\"/\"kube-root-ca.crt\"" Jul 24 23:01:02.412275 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.412256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:01:02.421668 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.421777 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.421777 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.421862 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.421928 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.421972 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.421960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9wf\" (UniqueName: \"kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.522970 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.522932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523148 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.522981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523148 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9wf\" (UniqueName: \"kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523148 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523148 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523148 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523421 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523478 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.523539 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.523497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.525485 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.525451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.525696 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.525677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.529873 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.529854 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9wf\" (UniqueName: \"kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf\") pod \"rate-test-model-e15986-kserve-7b4cfc6747-55rm2\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.715773 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.715678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:02.850632 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:02.850605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:01:02.853226 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:01:02.853201 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb65829f_6abf_4e69_bb93_d1d3c3748f6e.slice/crio-d740a964d3b0a22e6136b677cc6ead2ad30ad63a3997e658820a9487e730c03b WatchSource:0}: Error finding container d740a964d3b0a22e6136b677cc6ead2ad30ad63a3997e658820a9487e730c03b: Status 404 returned error can't find the container with id d740a964d3b0a22e6136b677cc6ead2ad30ad63a3997e658820a9487e730c03b Jul 24 23:01:03.648207 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:03.648167 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" event={"ID":"db65829f-6abf-4e69-bb93-d1d3c3748f6e","Type":"ContainerStarted","Data":"0cb0eb0721bbc78b50751a6475d6521c61961c4bd5a25a3b7bac3284f1c37a5f"} Jul 24 23:01:03.648728 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:03.648217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" event={"ID":"db65829f-6abf-4e69-bb93-d1d3c3748f6e","Type":"ContainerStarted","Data":"d740a964d3b0a22e6136b677cc6ead2ad30ad63a3997e658820a9487e730c03b"} Jul 24 23:01:03.663059 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:03.662992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" podStartSLOduration=1.662972875 podStartE2EDuration="1.662972875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:01:03.662401064 +0000 UTC m=+2786.329411885" watchObservedRunningTime="2026-07-24 23:01:03.662972875 +0000 UTC m=+2786.329983697" Jul 24 23:01:05.655126 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:05.655080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:01:12.716537 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:12.716504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:12.716955 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:12.716570 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:12.729322 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:12.729296 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:13.715418 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:13.715390 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:01:26.350409 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:26.350371 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:01:41.845245 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:41.845208 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:01:48.985994 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:48.985963 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:01:49.790230 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:49.790196 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:01:58.548844 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:01:58.548810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:02:04.006587 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.006527 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" podUID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" containerName="main" containerID="cri-o://cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2" gracePeriod=45 Jul 24 23:02:04.250186 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.250159 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:02:04.301735 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301668 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.301735 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.301892 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.301892 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.301892 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301859 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ctd9\" (UniqueName: \"kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.302036 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301900 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache\") pod \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\" (UID: \"49a519fe-4eeb-498f-8b33-98e7a034f0c2\") " Jul 24 23:02:04.302036 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.301917 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:04.302154 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.302107 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home" (OuterVolumeSpecName: "home") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:04.302222 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.302198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:04.302286 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.302272 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.302323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.302291 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.302323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.302301 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.303902 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.303880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:02:04.304468 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.304440 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm" (OuterVolumeSpecName: "dshm") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:04.304590 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.304469 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9" (OuterVolumeSpecName: "kube-api-access-7ctd9") pod "49a519fe-4eeb-498f-8b33-98e7a034f0c2" (UID: "49a519fe-4eeb-498f-8b33-98e7a034f0c2"). InnerVolumeSpecName "kube-api-access-7ctd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:02:04.403450 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.403426 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a519fe-4eeb-498f-8b33-98e7a034f0c2-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.403450 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.403446 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49a519fe-4eeb-498f-8b33-98e7a034f0c2-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.403624 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.403456 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctd9\" (UniqueName: \"kubernetes.io/projected/49a519fe-4eeb-498f-8b33-98e7a034f0c2-kube-api-access-7ctd9\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:04.809396 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.809326 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" podUID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" containerName="main" containerID="cri-o://0cb0eb0721bbc78b50751a6475d6521c61961c4bd5a25a3b7bac3284f1c37a5f" gracePeriod=45 Jul 24 23:02:04.949041 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.949014 2576 generic.go:356] "Generic (PLEG): container finished" podID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" containerID="0cb0eb0721bbc78b50751a6475d6521c61961c4bd5a25a3b7bac3284f1c37a5f" exitCode=0 Jul 24 23:02:04.949181 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.949086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" event={"ID":"db65829f-6abf-4e69-bb93-d1d3c3748f6e","Type":"ContainerDied","Data":"0cb0eb0721bbc78b50751a6475d6521c61961c4bd5a25a3b7bac3284f1c37a5f"} Jul 24 23:02:04.950488 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.950464 2576 generic.go:356] "Generic (PLEG): container finished" podID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" containerID="cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2" exitCode=0 Jul 24 23:02:04.950639 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.950505 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" Jul 24 23:02:04.950639 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.950506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" event={"ID":"49a519fe-4eeb-498f-8b33-98e7a034f0c2","Type":"ContainerDied","Data":"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2"} Jul 24 23:02:04.950639 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.950534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9" event={"ID":"49a519fe-4eeb-498f-8b33-98e7a034f0c2","Type":"ContainerDied","Data":"805549034bd12f1b76d1f88a18c3e069af2a20477d3eaf9b37c63f920164d919"} Jul 24 23:02:04.950639 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.950572 2576 scope.go:117] "RemoveContainer" containerID="cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2" Jul 24 23:02:04.960757 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.960652 2576 scope.go:117] "RemoveContainer" containerID="cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2" Jul 24 23:02:04.961027 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:02:04.960977 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2\": container with ID starting with cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2 not found: ID does not exist" containerID="cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2" Jul 24 23:02:04.961117 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.961028 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2"} err="failed to get container status \"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2\": rpc error: code = NotFound desc = could not find container \"cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2\": container with ID starting with cb067b8e8c8868f02a24f1cd789d3539a5274af4419d673cc601da137fdc19a2 not found: ID does not exist" Jul 24 23:02:04.974216 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.974193 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:02:04.978018 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:04.977995 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-499dd1/rate-test-model-499dd1-kserve-748899d7b7-52mv9"] Jul 24 23:02:05.063707 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.063658 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:02:05.211153 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211122 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc9wf\" (UniqueName: \"kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211282 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211319 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211472 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211336 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211472 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm\") pod \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\" (UID: \"db65829f-6abf-4e69-bb93-d1d3c3748f6e\") " Jul 24 23:02:05.211472 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211378 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:05.211661 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211517 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home" (OuterVolumeSpecName: "home") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:05.211661 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211625 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:05.211784 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211767 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.211835 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211789 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.211835 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.211804 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.213350 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.213330 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf" (OuterVolumeSpecName: "kube-api-access-vc9wf") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "kube-api-access-vc9wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:02:05.213942 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.213925 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm" (OuterVolumeSpecName: "dshm") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:02:05.214031 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.213940 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db65829f-6abf-4e69-bb93-d1d3c3748f6e" (UID: "db65829f-6abf-4e69-bb93-d1d3c3748f6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:02:05.312424 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.312379 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9wf\" (UniqueName: \"kubernetes.io/projected/db65829f-6abf-4e69-bb93-d1d3c3748f6e-kube-api-access-vc9wf\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.312424 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.312407 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db65829f-6abf-4e69-bb93-d1d3c3748f6e-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.312424 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.312415 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db65829f-6abf-4e69-bb93-d1d3c3748f6e-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:05.651520 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.651487 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:05.651914 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.651900 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" containerName="main" Jul 24 23:02:05.651959 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.651915 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" containerName="main" Jul 24 23:02:05.651959 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.651930 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" containerName="main" Jul 24 23:02:05.651959 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.651936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" containerName="main" Jul 24 23:02:05.652056 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.652002 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" containerName="main" Jul 24 23:02:05.652056 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.652009 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" containerName="main" Jul 24 23:02:05.654996 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.654980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.656974 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.656950 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vq9qt\"" Jul 24 23:02:05.658101 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.658079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:05.714693 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.714659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbnr\" (UniqueName: \"kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr\") pod \"test-curl-7537-139825557984720\" (UID: \"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.816093 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.816049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbnr\" (UniqueName: \"kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr\") pod \"test-curl-7537-139825557984720\" (UID: \"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.824150 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.824124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbnr\" (UniqueName: \"kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr\") pod \"test-curl-7537-139825557984720\" (UID: \"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee\") " pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.891657 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.891623 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a519fe-4eeb-498f-8b33-98e7a034f0c2" path="/var/lib/kubelet/pods/49a519fe-4eeb-498f-8b33-98e7a034f0c2/volumes" Jul 24 23:02:05.957960 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.957878 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" Jul 24 23:02:05.957960 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.957893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2" event={"ID":"db65829f-6abf-4e69-bb93-d1d3c3748f6e","Type":"ContainerDied","Data":"d740a964d3b0a22e6136b677cc6ead2ad30ad63a3997e658820a9487e730c03b"} Jul 24 23:02:05.957960 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.957932 2576 scope.go:117] "RemoveContainer" containerID="0cb0eb0721bbc78b50751a6475d6521c61961c4bd5a25a3b7bac3284f1c37a5f" Jul 24 23:02:05.965631 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.965612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:05.973269 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.973249 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:02:05.978021 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:05.978000 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e15986/rate-test-model-e15986-kserve-7b4cfc6747-55rm2"] Jul 24 23:02:06.101819 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:06.101791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:06.104323 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:02:06.104297 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6a4c89a_5dd8_40b5_b2e7_fe1524fdeeee.slice/crio-7de940ba8a18209d7ace98f48b008e5d070fc0484d849ef76d6a00200f222dd6 WatchSource:0}: Error finding container 7de940ba8a18209d7ace98f48b008e5d070fc0484d849ef76d6a00200f222dd6: Status 404 returned error can't find the container with id 7de940ba8a18209d7ace98f48b008e5d070fc0484d849ef76d6a00200f222dd6 Jul 24 23:02:06.105711 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:06.105689 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:02:06.964232 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:06.964204 2576 generic.go:356] "Generic (PLEG): container finished" podID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" containerID="b3d35a843f7cc24b55d8fe5e4b4dda8341296908149e9cfade6e87479ebab2fb" exitCode=0 Jul 24 23:02:06.964356 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:06.964283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825557984720" event={"ID":"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee","Type":"ContainerDied","Data":"b3d35a843f7cc24b55d8fe5e4b4dda8341296908149e9cfade6e87479ebab2fb"} Jul 24 23:02:06.964356 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:06.964319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-139825557984720" event={"ID":"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee","Type":"ContainerStarted","Data":"7de940ba8a18209d7ace98f48b008e5d070fc0484d849ef76d6a00200f222dd6"} Jul 24 23:02:07.895745 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:07.894165 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db65829f-6abf-4e69-bb93-d1d3c3748f6e" path="/var/lib/kubelet/pods/db65829f-6abf-4e69-bb93-d1d3c3748f6e/volumes" Jul 24 23:02:08.107332 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.107302 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:08.141521 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.141476 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7537-139825557984720_d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee/test-curl-7537-139825557984720/0.log" Jul 24 23:02:08.166602 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.166474 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:08.170035 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.170008 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-139825557984720"] Jul 24 23:02:08.238539 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.238506 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnbnr\" (UniqueName: \"kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr\") pod \"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee\" (UID: \"d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee\") " Jul 24 23:02:08.240914 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.240885 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr" (OuterVolumeSpecName: "kube-api-access-vnbnr") pod "d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" (UID: "d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee"). InnerVolumeSpecName "kube-api-access-vnbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:02:08.339961 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.339917 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbnr\" (UniqueName: \"kubernetes.io/projected/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee-kube-api-access-vnbnr\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:02:08.908702 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.908667 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:02:08.909089 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.909044 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" containerName="test-curl-7537-139825557984720" Jul 24 23:02:08.909089 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.909055 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" containerName="test-curl-7537-139825557984720" Jul 24 23:02:08.909173 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.909130 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" containerName="test-curl-7537-139825557984720" Jul 24 23:02:08.912206 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.912187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:08.914258 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.914231 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e3f0a9-openshift-default-dockercfg-54s2v\"" Jul 24 23:02:08.925166 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.925137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:02:08.977006 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.976979 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7de940ba8a18209d7ace98f48b008e5d070fc0484d849ef76d6a00200f222dd6" Jul 24 23:02:08.977185 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:08.977010 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:08.986696 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:02:08.986665 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-139825557984720\" is forbidden: User \"system:node:ip-10-0-132-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-132.ec2.internal' and this object" podUID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" pod="opendatahub/test-curl-7537-139825557984720" Jul 24 23:02:09.046090 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046090 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25sz\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046544 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.046544 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.046371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.146938 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.146902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.146951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.146978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147223 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25sz\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147678 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147678 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147810 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.147892 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.147868 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.148037 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.148017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.150052 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.150021 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.150217 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.150191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.154919 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.154886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.155031 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.155015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25sz\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz\") pod \"e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.224439 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.224337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:09.371354 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.371313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:02:09.374305 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:02:09.374275 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704fefd2_218d_4317_b808_eaa256820733.slice/crio-e1375e8eb3d044ba5e1bfa05f8e7ec20651c3ae9d5ce7355173bad1d4d25f49d WatchSource:0}: Error finding container e1375e8eb3d044ba5e1bfa05f8e7ec20651c3ae9d5ce7355173bad1d4d25f49d: Status 404 returned error can't find the container with id e1375e8eb3d044ba5e1bfa05f8e7ec20651c3ae9d5ce7355173bad1d4d25f49d Jul 24 23:02:09.376622 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.376592 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:09.376706 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.376654 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:09.376706 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.376684 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:09.890713 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.890681 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee" path="/var/lib/kubelet/pods/d6a4c89a-5dd8-40b5-b2e7-fe1524fdeeee/volumes" Jul 24 23:02:09.983731 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.983695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerStarted","Data":"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2"} Jul 24 23:02:09.984131 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:09.983746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerStarted","Data":"e1375e8eb3d044ba5e1bfa05f8e7ec20651c3ae9d5ce7355173bad1d4d25f49d"} Jul 24 23:02:10.002289 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:10.002231 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podStartSLOduration=2.002216382 podStartE2EDuration="2.002216382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:02:10.000712321 +0000 UTC m=+2852.667723155" watchObservedRunningTime="2026-07-24 23:02:10.002216382 +0000 UTC m=+2852.669227204" Jul 24 23:02:10.225143 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:10.225045 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:10.230820 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:10.230795 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:10.988758 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:10.988718 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:10.989631 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:10.989612 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:20.750963 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:20.750924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:02:27.030722 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.030683 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 23:02:27.031152 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.030737 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 23:02:27.515617 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.515582 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:02:27.530488 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.530461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.531782 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.531753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:02:27.532780 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.532760 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c6e95b-openshift-default-dockercfg-xjrjp\"" Jul 24 23:02:27.626029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.625992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626293 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626293 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626293 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665rv\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626293 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626293 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626511 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.626511 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.626346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.726953 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.726914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.726953 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.726958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727292 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.726987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727292 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727292 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727292 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727502 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727583 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727583 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665rv\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727690 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.727872 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.728081 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.728055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.728249 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.727960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.729907 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.729876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.730003 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.729958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.734733 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.734698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.734895 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.734881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665rv\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv\") pod \"e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:27.845228 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:27.845140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:28.026412 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:28.026369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:02:28.027338 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:02:28.027310 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da1828a_a0db_4323_9f6d_5751c560ee60.slice/crio-94aa96efe02506b7cb22703d7d62a64ea2c83c4a21314fe493e8b3d4940c0386 WatchSource:0}: Error finding container 94aa96efe02506b7cb22703d7d62a64ea2c83c4a21314fe493e8b3d4940c0386: Status 404 returned error can't find the container with id 94aa96efe02506b7cb22703d7d62a64ea2c83c4a21314fe493e8b3d4940c0386 Jul 24 23:02:28.029990 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:28.029947 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:28.030097 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:28.030027 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:28.030097 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:28.030073 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:28.081272 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:28.081238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerStarted","Data":"94aa96efe02506b7cb22703d7d62a64ea2c83c4a21314fe493e8b3d4940c0386"} Jul 24 23:02:29.087545 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.087454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerStarted","Data":"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe"} Jul 24 23:02:29.088939 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.088912 2576 generic.go:356] "Generic (PLEG): container finished" podID="704fefd2-218d-4317-b808-eaa256820733" containerID="1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2" exitCode=0 Jul 24 23:02:29.089076 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.088950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerDied","Data":"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2"} Jul 24 23:02:29.089326 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.089309 2576 scope.go:117] "RemoveContainer" containerID="1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2" Jul 24 23:02:29.090339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.090312 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:29.090424 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.090379 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:29.090424 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.090408 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:29.108071 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.108024 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podStartSLOduration=2.108007521 podStartE2EDuration="2.108007521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:02:29.106136499 +0000 UTC m=+2871.773147322" watchObservedRunningTime="2026-07-24 23:02:29.108007521 +0000 UTC m=+2871.775018345" Jul 24 23:02:29.846246 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.846196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:29.851776 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:29.851750 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:30.095901 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.095867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerStarted","Data":"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73"} Jul 24 23:02:30.096391 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.096024 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:30.096937 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.096920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:30.225287 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.225238 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:30.225479 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.225456 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:02:30.225543 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:30.225516 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 24 23:02:31.224948 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:31.224915 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:02:31.225360 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:31.224980 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 24 23:02:32.225187 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:32.225149 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:02:32.225697 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:32.225214 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 24 23:02:33.224840 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:33.224798 2576 patch_prober.go:28] interesting pod/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 24 23:02:33.225033 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:33.224854 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 24 23:02:34.246323 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:34.246279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:34.246824 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:34.246644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:34.247752 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:34.247702 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:02:40.146256 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:40.146222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:02:46.138765 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:46.138715 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 23:02:46.139188 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:46.138819 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 23:02:48.194954 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.194877 2576 generic.go:356] "Generic (PLEG): container finished" podID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerID="95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe" exitCode=0 Jul 24 23:02:48.195365 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.194963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerDied","Data":"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe"} Jul 24 23:02:48.195365 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.195343 2576 scope.go:117] "RemoveContainer" containerID="95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe" Jul 24 23:02:48.196398 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.196373 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:48.196461 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.196440 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:48.196502 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:48.196472 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:02:49.202095 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:49.202064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerStarted","Data":"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163"} Jul 24 23:02:49.845996 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:49.845901 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:49.846645 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:49.846619 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 24 23:02:49.846882 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:49.846855 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 24 23:02:50.845634 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:50.845594 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 24 23:02:50.846155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:50.845669 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 24 23:02:51.846658 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:51.846618 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 24 23:02:51.847257 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:51.847224 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 24 23:02:52.846353 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:52.846316 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 24 23:02:52.846680 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:52.846649 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 24 23:02:53.848894 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:53.848863 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:53.849274 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:53.849218 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:02:53.849834 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:02:53.849809 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:03:03.696917 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.696883 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:03:03.700231 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.700209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.702206 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.702184 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e3f0a9\"/\"openshift-service-ca.crt\"" Jul 24 23:03:03.702787 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.702759 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e3f0a9\"/\"kube-root-ca.crt\"" Jul 24 23:03:03.702900 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.702792 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e3f0a9\"/\"default-dockercfg-q52v6\"" Jul 24 23:03:03.702900 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.702833 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e3f0a9\"/\"ipp-route-e3f0a9-kserve-self-signed-certs\"" Jul 24 23:03:03.711579 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.711530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:03:03.798098 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.798098 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.798299 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5stj\" (UniqueName: \"kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.798299 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.798299 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.798299 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.798248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898771 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.898945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.898910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5stj\" (UniqueName: \"kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.899212 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.899189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.899275 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.899207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.899275 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.899244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.901295 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.901274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.901604 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.901586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:03.906851 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:03.906831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5stj\" (UniqueName: \"kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj\") pod \"ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:04.010159 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:04.010068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:04.145978 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:04.145929 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:03:04.148256 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:03:04.148228 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0cf6a29_6699_4c94_8445_30a4908ba9c2.slice/crio-31a7f151bc8a76d1bb76f868a6bc60fdab38e5464855546838c6fef22b6f8ae0 WatchSource:0}: Error finding container 31a7f151bc8a76d1bb76f868a6bc60fdab38e5464855546838c6fef22b6f8ae0: Status 404 returned error can't find the container with id 31a7f151bc8a76d1bb76f868a6bc60fdab38e5464855546838c6fef22b6f8ae0 Jul 24 23:03:04.274938 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:04.274851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" event={"ID":"c0cf6a29-6699-4c94-8445-30a4908ba9c2","Type":"ContainerStarted","Data":"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d"} Jul 24 23:03:04.274938 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:04.274896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" event={"ID":"c0cf6a29-6699-4c94-8445-30a4908ba9c2","Type":"ContainerStarted","Data":"31a7f151bc8a76d1bb76f868a6bc60fdab38e5464855546838c6fef22b6f8ae0"} Jul 24 23:03:04.291258 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:04.291201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" podStartSLOduration=1.291185908 podStartE2EDuration="1.291185908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:03:04.29011597 +0000 UTC m=+2906.957126791" watchObservedRunningTime="2026-07-24 23:03:04.291185908 +0000 UTC m=+2906.958196729" Jul 24 23:03:07.451809 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:07.451774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:03:14.011067 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:14.011029 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:14.011595 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:14.011122 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:14.024124 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:14.024100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:14.334493 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:14.334409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:03:22.655644 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:22.655604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:03:34.965996 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:34.965962 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:03:34.969744 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:34.969719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:34.971985 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:34.971965 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-34f01f-openshift-default-dockercfg-jwmw8\"" Jul 24 23:03:34.981377 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:34.981345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:03:35.078528 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.078764 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcjv\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.078764 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.078764 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.079032 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.079032 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.079032 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.079032 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.079032 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.078933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180361 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180361 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcjv\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180662 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180938 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.180938 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.181046 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.180968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.181102 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.181042 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.181396 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.181376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.182991 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.182971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.183311 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.183294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.189735 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.189710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.189875 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.189767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcjv\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv\") pod \"e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.281366 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.281274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:35.423806 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.423775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:03:35.425973 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:03:35.425944 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9efb9c_a62e_451e_bfd7_b2630132b439.slice/crio-2baf235715b4c295bed4745cf01edd40b14c0fbfa164d04147050fd56887a3bf WatchSource:0}: Error finding container 2baf235715b4c295bed4745cf01edd40b14c0fbfa164d04147050fd56887a3bf: Status 404 returned error can't find the container with id 2baf235715b4c295bed4745cf01edd40b14c0fbfa164d04147050fd56887a3bf Jul 24 23:03:35.431009 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.430979 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:35.431096 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.431044 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:35.431096 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:35.431071 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:36.431726 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:36.431689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerStarted","Data":"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e"} Jul 24 23:03:36.432109 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:36.431742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerStarted","Data":"2baf235715b4c295bed4745cf01edd40b14c0fbfa164d04147050fd56887a3bf"} Jul 24 23:03:36.454405 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:36.454346 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podStartSLOduration=2.454326608 podStartE2EDuration="2.454326608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:03:36.451128635 +0000 UTC m=+2939.118139457" watchObservedRunningTime="2026-07-24 23:03:36.454326608 +0000 UTC m=+2939.121337434" Jul 24 23:03:37.281772 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:37.281741 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:37.286671 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:37.286645 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:37.436174 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:37.436142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:37.437162 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:37.437139 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:42.250016 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:42.249979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:03:50.509965 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.509921 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerID="28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e" exitCode=0 Jul 24 23:03:50.510440 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.510065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerDied","Data":"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e"} Jul 24 23:03:50.510776 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.510753 2576 scope.go:117] "RemoveContainer" containerID="28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e" Jul 24 23:03:50.512603 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.512548 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:50.512696 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.512657 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:50.512745 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:50.512698 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 24 23:03:51.516647 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:51.516614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerStarted","Data":"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c"} Jul 24 23:03:52.282315 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:52.282274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:52.283364 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:52.283338 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:03:52.283477 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:52.283391 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 23:03:52.436242 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:52.436207 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:53.282288 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:53.282256 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:03:53.282708 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:53.282309 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 23:03:54.281828 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:54.281787 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:03:54.282012 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:54.281861 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 23:03:55.281846 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:55.281809 2576 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 23:03:55.282276 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:55.281865 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 23:03:56.286195 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:56.286166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:03:56.287252 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:03:56.287229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:04:08.753371 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:08.753289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:04:15.039297 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:15.039261 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:04:15.039769 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:15.039478 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" containerID="cri-o://c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c" gracePeriod=30 Jul 24 23:04:20.298304 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.298240 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:04:20.406630 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406630 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406639 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406665 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406709 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjcjv\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406774 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406810 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.406934 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert\") pod \"9c9efb9c-a62e-451e-bfd7-b2630132b439\" (UID: \"9c9efb9c-a62e-451e-bfd7-b2630132b439\") " Jul 24 23:04:20.407244 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.406964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:20.407244 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407041 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:20.407244 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407191 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.407244 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407209 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-workload-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.407441 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407243 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:20.407441 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:04:20.407548 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.407437 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:20.409324 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.409288 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:04:20.409443 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.409366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:20.409443 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.409378 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv" (OuterVolumeSpecName: "kube-api-access-tjcjv") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "kube-api-access-tjcjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:20.409443 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.409379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c9efb9c-a62e-451e-bfd7-b2630132b439" (UID: "9c9efb9c-a62e-451e-bfd7-b2630132b439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:20.508047 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508024 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-data\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508047 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508046 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-podinfo\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508210 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508056 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcjv\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-kube-api-access-tjcjv\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508210 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508064 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-token\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508210 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508072 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-credential-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508210 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508081 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9efb9c-a62e-451e-bfd7-b2630132b439-istio-envoy\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.508210 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.508088 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9efb9c-a62e-451e-bfd7-b2630132b439-istiod-ca-cert\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:20.657474 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.657442 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerID="c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c" exitCode=0 Jul 24 23:04:20.657622 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.657496 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" Jul 24 23:04:20.657622 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.657541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerDied","Data":"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c"} Jul 24 23:04:20.657708 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.657624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl" event={"ID":"9c9efb9c-a62e-451e-bfd7-b2630132b439","Type":"ContainerDied","Data":"2baf235715b4c295bed4745cf01edd40b14c0fbfa164d04147050fd56887a3bf"} Jul 24 23:04:20.657708 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.657653 2576 scope.go:117] "RemoveContainer" containerID="c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c" Jul 24 23:04:20.667155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.667139 2576 scope.go:117] "RemoveContainer" containerID="28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e" Jul 24 23:04:20.675215 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.675199 2576 scope.go:117] "RemoveContainer" containerID="c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c" Jul 24 23:04:20.675462 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:04:20.675444 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c\": container with ID starting with c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c not found: ID does not exist" containerID="c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c" Jul 24 23:04:20.675517 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.675469 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c"} err="failed to get container status \"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c\": rpc error: code = NotFound desc = could not find container \"c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c\": container with ID starting with c50b60077de0c1afb661b38fd4edec59c23e6d4906905578d1ae5c0094b6139c not found: ID does not exist" Jul 24 23:04:20.675517 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.675486 2576 scope.go:117] "RemoveContainer" containerID="28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e" Jul 24 23:04:20.675823 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:04:20.675804 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e\": container with ID starting with 28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e not found: ID does not exist" containerID="28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e" Jul 24 23:04:20.675868 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.675829 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e"} err="failed to get container status \"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e\": rpc error: code = NotFound desc = could not find container \"28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e\": container with ID starting with 28334d26efb47f4c3524485a2c754f68ec89b4f9ffa1418da903bd337affce9e not found: ID does not exist" Jul 24 23:04:20.679080 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.679060 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:04:20.686587 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:20.683963 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-34f01f-openshift-default-5ff59f98d8-c6nkl"] Jul 24 23:04:21.890105 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:21.890069 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" path="/var/lib/kubelet/pods/9c9efb9c-a62e-451e-bfd7-b2630132b439/volumes" Jul 24 23:04:35.455745 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:35.455697 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:04:44.049684 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:44.049647 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:04:46.246127 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:46.246088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:04:59.069606 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.069494 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" podUID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" containerName="main" containerID="cri-o://5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d" gracePeriod=45 Jul 24 23:04:59.324986 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.324932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:04:59.478336 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478336 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478343 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478608 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478608 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478716 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478624 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478716 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478643 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:59.478716 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478661 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:59.478716 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5stj\" (UniqueName: \"kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj\") pod \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\" (UID: \"c0cf6a29-6699-4c94-8445-30a4908ba9c2\") " Jul 24 23:04:59.478912 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478815 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home" (OuterVolumeSpecName: "home") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:59.478992 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478977 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-model-cache\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.479200 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.478994 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tmp-dir\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.479200 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.479002 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-home\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.480760 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.480742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm" (OuterVolumeSpecName: "dshm") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:04:59.480860 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.480802 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:04:59.480860 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.480823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj" (OuterVolumeSpecName: "kube-api-access-s5stj") pod "c0cf6a29-6699-4c94-8445-30a4908ba9c2" (UID: "c0cf6a29-6699-4c94-8445-30a4908ba9c2"). InnerVolumeSpecName "kube-api-access-s5stj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:04:59.580059 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.579987 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5stj\" (UniqueName: \"kubernetes.io/projected/c0cf6a29-6699-4c94-8445-30a4908ba9c2-kube-api-access-s5stj\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.580059 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.580014 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0cf6a29-6699-4c94-8445-30a4908ba9c2-tls-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.580059 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.580024 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0cf6a29-6699-4c94-8445-30a4908ba9c2-dshm\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:04:59.846173 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.846083 2576 generic.go:356] "Generic (PLEG): container finished" podID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" containerID="5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d" exitCode=0 Jul 24 23:04:59.846173 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.846127 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" Jul 24 23:04:59.846362 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.846168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" event={"ID":"c0cf6a29-6699-4c94-8445-30a4908ba9c2","Type":"ContainerDied","Data":"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d"} Jul 24 23:04:59.846362 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.846213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6" event={"ID":"c0cf6a29-6699-4c94-8445-30a4908ba9c2","Type":"ContainerDied","Data":"31a7f151bc8a76d1bb76f868a6bc60fdab38e5464855546838c6fef22b6f8ae0"} Jul 24 23:04:59.846362 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.846230 2576 scope.go:117] "RemoveContainer" containerID="5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d" Jul 24 23:04:59.855466 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.855444 2576 scope.go:117] "RemoveContainer" containerID="5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d" Jul 24 23:04:59.855754 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:04:59.855736 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d\": container with ID starting with 5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d not found: ID does not exist" containerID="5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d" Jul 24 23:04:59.855822 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.855762 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d"} err="failed to get container status \"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d\": rpc error: code = NotFound desc = could not find container \"5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d\": container with ID starting with 5115e4d922a96844189b6f1290608719ddfa2c0ce9751e877dc5d57fe7c63b7d not found: ID does not exist" Jul 24 23:04:59.866440 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.866418 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:04:59.870049 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.870029 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e3f0a9/ipp-route-e3f0a9-kserve-6b6b5456cf-tnsp6"] Jul 24 23:04:59.890739 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:04:59.890718 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" path="/var/lib/kubelet/pods/c0cf6a29-6699-4c94-8445-30a4908ba9c2/volumes" Jul 24 23:05:27.142205 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:27.142169 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:05:27.142804 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:27.142432 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" containerID="cri-o://f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73" gracePeriod=30 Jul 24 23:05:32.407796 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.407741 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:05:32.490981 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.490955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.490990 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491021 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491059 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491155 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491433 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491170 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491433 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491200 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25sz\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491433 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491266 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo\") pod \"704fefd2-218d-4317-b808-eaa256820733\" (UID: \"704fefd2-218d-4317-b808-eaa256820733\") " Jul 24 23:05:32.491433 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491290 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:32.491670 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491503 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:32.491725 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491686 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-credential-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.491725 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:32.491725 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491693 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:32.491725 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491707 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.491982 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.491961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data" (OuterVolumeSpecName: "istio-data") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:32.493735 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.493708 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:05:32.493847 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.493807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz" (OuterVolumeSpecName: "kube-api-access-k25sz") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "kube-api-access-k25sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:32.493985 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.493971 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token" (OuterVolumeSpecName: "istio-token") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:32.494030 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.493974 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "704fefd2-218d-4317-b808-eaa256820733" (UID: "704fefd2-218d-4317-b808-eaa256820733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:32.592786 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592758 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704fefd2-218d-4317-b808-eaa256820733-istiod-ca-cert\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.592786 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592782 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-workload-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.592786 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592791 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25sz\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-kube-api-access-k25sz\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.593029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592800 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704fefd2-218d-4317-b808-eaa256820733-istio-podinfo\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.593029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592810 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704fefd2-218d-4317-b808-eaa256820733-istio-token\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.593029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592818 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-data\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:32.593029 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:32.592826 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704fefd2-218d-4317-b808-eaa256820733-istio-envoy\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:33.004698 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.004666 2576 generic.go:356] "Generic (PLEG): container finished" podID="704fefd2-218d-4317-b808-eaa256820733" containerID="f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73" exitCode=0 Jul 24 23:05:33.004861 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.004705 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" Jul 24 23:05:33.004861 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.004746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerDied","Data":"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73"} Jul 24 23:05:33.004861 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.004797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m" event={"ID":"704fefd2-218d-4317-b808-eaa256820733","Type":"ContainerDied","Data":"e1375e8eb3d044ba5e1bfa05f8e7ec20651c3ae9d5ce7355173bad1d4d25f49d"} Jul 24 23:05:33.004861 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.004826 2576 scope.go:117] "RemoveContainer" containerID="f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73" Jul 24 23:05:33.014574 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.014544 2576 scope.go:117] "RemoveContainer" containerID="1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2" Jul 24 23:05:33.022530 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.022510 2576 scope.go:117] "RemoveContainer" containerID="f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73" Jul 24 23:05:33.022804 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:05:33.022784 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73\": container with ID starting with f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73 not found: ID does not exist" containerID="f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73" Jul 24 23:05:33.022845 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.022814 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73"} err="failed to get container status \"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73\": rpc error: code = NotFound desc = could not find container \"f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73\": container with ID starting with f21cfe2a6d8a4e65dfce9837615b4182864ff309e46839fa2a6a76ff081b1b73 not found: ID does not exist" Jul 24 23:05:33.022845 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.022831 2576 scope.go:117] "RemoveContainer" containerID="1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2" Jul 24 23:05:33.023236 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:05:33.023131 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2\": container with ID starting with 1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2 not found: ID does not exist" containerID="1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2" Jul 24 23:05:33.023236 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.023172 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2"} err="failed to get container status \"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2\": rpc error: code = NotFound desc = could not find container \"1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2\": container with ID starting with 1c9fcf37913e7f3a48d8955b9a680c8ab8b008379f3bb25ff17224c0625037a2 not found: ID does not exist" Jul 24 23:05:33.025474 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.025455 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:05:33.029039 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.029020 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e3f0a9-openshift-default-8588586558-2br4m"] Jul 24 23:05:33.890875 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:33.890840 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704fefd2-218d-4317-b808-eaa256820733" path="/var/lib/kubelet/pods/704fefd2-218d-4317-b808-eaa256820733/volumes" Jul 24 23:05:40.247729 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:40.247691 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:05:48.847120 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:48.847077 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:05:48.847590 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:48.847293 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" containerID="cri-o://8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163" gracePeriod=30 Jul 24 23:05:53.855121 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:53.855082 2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": EOF" start-of-body= Jul 24 23:05:53.855575 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:53.855159 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": EOF" Jul 24 23:05:54.101260 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.101236 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:05:54.106846 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.106778 2576 generic.go:356] "Generic (PLEG): container finished" podID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerID="8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163" exitCode=0 Jul 24 23:05:54.106846 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.106825 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" Jul 24 23:05:54.107014 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.106857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerDied","Data":"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163"} Jul 24 23:05:54.107014 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.106914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r" event={"ID":"7da1828a-a0db-4323-9f6d-5751c560ee60","Type":"ContainerDied","Data":"94aa96efe02506b7cb22703d7d62a64ea2c83c4a21314fe493e8b3d4940c0386"} Jul 24 23:05:54.107014 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.106938 2576 scope.go:117] "RemoveContainer" containerID="8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163" Jul 24 23:05:54.115596 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.115577 2576 scope.go:117] "RemoveContainer" containerID="95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe" Jul 24 23:05:54.123451 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.123430 2576 scope.go:117] "RemoveContainer" containerID="8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163" Jul 24 23:05:54.123736 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:05:54.123718 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163\": container with ID starting with 8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163 not found: ID does not exist" containerID="8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163" Jul 24 23:05:54.123799 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.123745 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163"} err="failed to get container status \"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163\": rpc error: code = NotFound desc = could not find container \"8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163\": container with ID starting with 8bc56a0a4f7b898abb85b8ef7e4395a8fd46940fa3a6976edb3d442faa2eb163 not found: ID does not exist" Jul 24 23:05:54.123799 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.123761 2576 scope.go:117] "RemoveContainer" containerID="95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe" Jul 24 23:05:54.124020 ip-10-0-132-132 kubenswrapper[2576]: E0724 23:05:54.124006 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe\": container with ID starting with 95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe not found: ID does not exist" containerID="95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe" Jul 24 23:05:54.124062 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.124025 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe"} err="failed to get container status \"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe\": rpc error: code = NotFound desc = could not find container \"95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe\": container with ID starting with 95931a96948d5e21414efc8233dbf3917e738c7ee7a289d92812216e2f5388fe not found: ID does not exist" Jul 24 23:05:54.184171 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-665rv\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184236 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184339 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184501 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184407 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184501 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184455 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184501 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184462 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:54.184501 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184482 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184737 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184737 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token\") pod \"7da1828a-a0db-4323-9f6d-5751c560ee60\" (UID: \"7da1828a-a0db-4323-9f6d-5751c560ee60\") " Jul 24 23:05:54.184737 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184666 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:54.184908 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.184778 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:54.185079 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.185060 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:05:54.185164 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.185145 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-certs\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.185212 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.185170 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-credential-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.185212 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.185185 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-workload-socket\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.185511 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.185475 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data" (OuterVolumeSpecName: "istio-data") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:54.186609 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.186584 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv" (OuterVolumeSpecName: "kube-api-access-665rv") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "kube-api-access-665rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:54.186772 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.186747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:05:54.187119 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.187098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:05:54.187174 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.187099 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token" (OuterVolumeSpecName: "istio-token") pod "7da1828a-a0db-4323-9f6d-5751c560ee60" (UID: "7da1828a-a0db-4323-9f6d-5751c560ee60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:05:54.285997 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.285961 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-token\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.285997 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.285992 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665rv\" (UniqueName: \"kubernetes.io/projected/7da1828a-a0db-4323-9f6d-5751c560ee60-kube-api-access-665rv\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.285997 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.286005 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-data\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.286243 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.286019 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-podinfo\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.286243 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.286046 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7da1828a-a0db-4323-9f6d-5751c560ee60-istio-envoy\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.286243 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.286058 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7da1828a-a0db-4323-9f6d-5751c560ee60-istiod-ca-cert\") on node \"ip-10-0-132-132.ec2.internal\" DevicePath \"\"" Jul 24 23:05:54.428601 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.428570 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:05:54.431975 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:54.431948 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c6e95b-openshift-default-7c655cbb96-tmj4r"] Jul 24 23:05:55.889986 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:05:55.889952 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" path="/var/lib/kubelet/pods/7da1828a-a0db-4323-9f6d-5751c560ee60/volumes" Jul 24 23:06:02.558476 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:02.558437 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:06:24.349437 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:24.349398 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8c565"] Jul 24 23:06:38.458121 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:38.458045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rlqln_f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e/manager/0.log" Jul 24 23:06:39.033073 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:39.033046 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-99577_f7df0e12-4115-4f18-aa32-98434430ba37/copy-manifests/0.log" Jul 24 23:06:39.040300 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:39.040280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-99577_f7df0e12-4115-4f18-aa32-98434430ba37/manager/0.log" Jul 24 23:06:41.837912 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:41.837883 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8c565_084fe24f-935e-44a4-a254-c60909f6af3d/limitador/0.log" Jul 24 23:06:42.518858 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:42.518825 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-c45m2_bbad459a-837d-46ed-875b-c241c1d64072/kube-auth-proxy/0.log" Jul 24 23:06:42.747780 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:42.747747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/1.log" Jul 24 23:06:43.641099 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:43.641067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ggfdd_a7d47a34-9a37-447f-8327-7ca622cf14ff/main/0.log" Jul 24 23:06:43.859499 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:43.859468 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wbpfh_c161d106-1fbf-4da9-ba00-ddbd589a34e8/main/0.log" Jul 24 23:06:43.962200 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:43.962120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-v844f_9cd3732c-0934-4301-9657-9ec3029c2b49/main/0.log" Jul 24 23:06:51.848331 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:51.848294 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vpxvh_dc3a409d-d52b-4761-901f-a7a4c6ca7ca4/global-pull-secret-syncer/0.log" Jul 24 23:06:51.995288 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:51.995256 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zzhmd_9943b7f9-8c27-4004-ab2b-f21eeef6bf73/konnectivity-agent/0.log" Jul 24 23:06:52.014327 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:52.014299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-132.ec2.internal_1eb745cffc865678816fc0718fd4a2e2/haproxy/0.log" Jul 24 23:06:58.300232 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:06:58.300202 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8c565_084fe24f-935e-44a4-a254-c60909f6af3d/limitador/0.log" Jul 24 23:07:00.278997 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.278968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78978c6859-fb6rg_6b7dec87-0f5b-472c-8aea-1d7b02338b7c/metrics-server/0.log" Jul 24 23:07:00.300582 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.299923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-9zbvv_0dfae74e-ccb6-4393-8ef0-20459106cb1a/monitoring-plugin/0.log" Jul 24 23:07:00.443929 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.443904 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsvk_80ccf359-893c-4094-bd87-ca216bb2ec75/node-exporter/0.log" Jul 24 23:07:00.459772 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.459751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsvk_80ccf359-893c-4094-bd87-ca216bb2ec75/kube-rbac-proxy/0.log" Jul 24 23:07:00.476833 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.476812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfsvk_80ccf359-893c-4094-bd87-ca216bb2ec75/init-textfile/0.log" Jul 24 23:07:00.495525 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.495508 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nzmtg_bdd5fb11-6bb2-4c2e-9406-0f22a90bb803/kube-rbac-proxy-main/0.log" Jul 24 23:07:00.510573 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.510534 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nzmtg_bdd5fb11-6bb2-4c2e-9406-0f22a90bb803/kube-rbac-proxy-self/0.log" Jul 24 23:07:00.531120 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.531101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-nzmtg_bdd5fb11-6bb2-4c2e-9406-0f22a90bb803/openshift-state-metrics/0.log" Jul 24 23:07:00.556141 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.556095 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/prometheus/0.log" Jul 24 23:07:00.571705 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.571688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/config-reloader/0.log" Jul 24 23:07:00.591187 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.591168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/thanos-sidecar/0.log" Jul 24 23:07:00.606281 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.606263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/kube-rbac-proxy-web/0.log" Jul 24 23:07:00.622025 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.622002 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/kube-rbac-proxy/0.log" Jul 24 23:07:00.639080 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.639060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/kube-rbac-proxy-thanos/0.log" Jul 24 23:07:00.654671 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.654647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c112db9d-4fc7-4532-bbbd-eda08af07e92/init-config-reloader/0.log" Jul 24 23:07:00.711695 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.711635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-5jkw6_7f412ff1-111f-4042-a67d-7033337d2fb4/prometheus-operator-admission-webhook/0.log" Jul 24 23:07:00.806436 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.806413 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/thanos-query/0.log" Jul 24 23:07:00.821663 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.821641 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/kube-rbac-proxy-web/0.log" Jul 24 23:07:00.837526 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.837500 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/kube-rbac-proxy/0.log" Jul 24 23:07:00.852567 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.852530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/prom-label-proxy/0.log" Jul 24 23:07:00.867722 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.867705 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/kube-rbac-proxy-rules/0.log" Jul 24 23:07:00.883340 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:00.883323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b84699cd5-fqqjk_c0666bc7-fe4a-4e2b-ae81-1ceb823561ee/kube-rbac-proxy-metrics/0.log" Jul 24 23:07:01.998125 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:01.998091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-jpnv6_79b7ae50-984d-4554-b074-afa345084313/networking-console-plugin/0.log" Jul 24 23:07:02.023404 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023377 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x"] Jul 24 23:07:02.023781 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023768 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023783 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023801 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023807 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023814 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023821 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" Jul 24 23:07:02.023830 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023829 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" containerName="main" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023835 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" containerName="main" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023898 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0cf6a29-6699-4c94-8445-30a4908ba9c2" containerName="main" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023905 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023911 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023917 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9efb9c-a62e-451e-bfd7-b2630132b439" containerName="istio-proxy" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023924 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da1828a-a0db-4323-9f6d-5751c560ee60" containerName="istio-proxy" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023930 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" Jul 24 23:07:02.024015 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.023936 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="704fefd2-218d-4317-b808-eaa256820733" containerName="istio-proxy" Jul 24 23:07:02.027123 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.027102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.028843 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.028824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsljv\"/\"kube-root-ca.crt\"" Jul 24 23:07:02.028945 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.028864 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsljv\"/\"openshift-service-ca.crt\"" Jul 24 23:07:02.028996 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.028963 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rsljv\"/\"default-dockercfg-66cxv\"" Jul 24 23:07:02.035779 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.035757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x"] Jul 24 23:07:02.098462 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.098439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-sys\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.098586 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.098472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdd6\" (UniqueName: \"kubernetes.io/projected/554c1d16-40af-4f30-a779-1fe68a684fab-kube-api-access-4vdd6\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.098586 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.098503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-lib-modules\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.098586 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.098547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-proc\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.098711 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.098630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-podres\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199775 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-proc\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199885 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-podres\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199885 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-sys\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199885 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdd6\" (UniqueName: \"kubernetes.io/projected/554c1d16-40af-4f30-a779-1fe68a684fab-kube-api-access-4vdd6\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199885 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-proc\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.199885 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-lib-modules\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.200070 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-sys\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.200070 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-lib-modules\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.200070 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.199993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/554c1d16-40af-4f30-a779-1fe68a684fab-podres\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.207405 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.207383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdd6\" (UniqueName: \"kubernetes.io/projected/554c1d16-40af-4f30-a779-1fe68a684fab-kube-api-access-4vdd6\") pod \"perf-node-gather-daemonset-6r86x\" (UID: \"554c1d16-40af-4f30-a779-1fe68a684fab\") " pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.338052 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.337991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:02.473252 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:02.473225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x"] Jul 24 23:07:02.475800 ip-10-0-132-132 kubenswrapper[2576]: W0724 23:07:02.475775 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod554c1d16_40af_4f30_a779_1fe68a684fab.slice/crio-b3a47b637815325be44eb3fbfe616f2b53544dd95285addc48b47d451065dd0f WatchSource:0}: Error finding container b3a47b637815325be44eb3fbfe616f2b53544dd95285addc48b47d451065dd0f: Status 404 returned error can't find the container with id b3a47b637815325be44eb3fbfe616f2b53544dd95285addc48b47d451065dd0f Jul 24 23:07:03.220144 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.220097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-dpwjh_dea03dea-67ef-4cc9-93ec-cde8df14463d/download-server/0.log" Jul 24 23:07:03.433868 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.433833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" event={"ID":"554c1d16-40af-4f30-a779-1fe68a684fab","Type":"ContainerStarted","Data":"b1b31933b3d06b7395e281d748eba58a07276c0ef280260b9e3b3763aa022e32"} Jul 24 23:07:03.434034 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.433882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" event={"ID":"554c1d16-40af-4f30-a779-1fe68a684fab","Type":"ContainerStarted","Data":"b3a47b637815325be44eb3fbfe616f2b53544dd95285addc48b47d451065dd0f"} Jul 24 23:07:03.434034 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.433963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:03.448926 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.448880 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" podStartSLOduration=1.448863759 podStartE2EDuration="1.448863759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:07:03.447525745 +0000 UTC m=+3146.114536572" watchObservedRunningTime="2026-07-24 23:07:03.448863759 +0000 UTC m=+3146.115874575" Jul 24 23:07:03.783123 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:03.783091 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-cmpjr_8a40dd43-2cff-4f53-ad6e-d8deefb954ba/volume-data-source-validator/0.log" Jul 24 23:07:04.772845 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:04.772809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gmhgc_08c4a51f-148d-4a9f-a9cd-dee108d95b40/dns/0.log" Jul 24 23:07:04.794160 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:04.794129 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gmhgc_08c4a51f-148d-4a9f-a9cd-dee108d95b40/kube-rbac-proxy/0.log" Jul 24 23:07:04.873774 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:04.873751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zjk2_f0824b1c-cbf0-4438-afa9-b628cb1b4203/dns-node-resolver/0.log" Jul 24 23:07:05.443017 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:05.442982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjl4j_0a3330cf-b271-47f3-8cb8-80eea9cfaa50/node-ca/0.log" Jul 24 23:07:06.493068 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:06.493043 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-c45m2_bbad459a-837d-46ed-875b-c241c1d64072/kube-auth-proxy/0.log" Jul 24 23:07:06.535203 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:06.535179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/0.log" Jul 24 23:07:06.562823 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:06.562788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/1.log" Jul 24 23:07:07.433813 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:07.433785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8c4mn_e8cb2a34-e342-4afa-8915-ac6efad06772/serve-healthcheck-canary/0.log" Jul 24 23:07:07.985477 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:07.985444 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzzjg_b983e892-a4d5-4ed3-9e28-4bce97b48431/kube-rbac-proxy/0.log" Jul 24 23:07:08.000953 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:08.000926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzzjg_b983e892-a4d5-4ed3-9e28-4bce97b48431/exporter/0.log" Jul 24 23:07:08.016449 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:08.016434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzzjg_b983e892-a4d5-4ed3-9e28-4bce97b48431/extractor/0.log" Jul 24 23:07:09.449245 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:09.449218 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rsljv/perf-node-gather-daemonset-6r86x" Jul 24 23:07:10.270657 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:10.270624 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rlqln_f5f1ade1-5d6d-463f-a9f9-0c9fa7e9be7e/manager/0.log" Jul 24 23:07:10.596375 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:10.596289 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-99577_f7df0e12-4115-4f18-aa32-98434430ba37/manager/0.log" Jul 24 23:07:10.611658 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:10.611636 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854fb8676-99577_f7df0e12-4115-4f18-aa32-98434430ba37/copy-manifests/0.log" Jul 24 23:07:12.120287 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:12.120261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b958b6f7b-2dx7b_29a00523-947f-45ae-92b5-98d7e3ef25c0/manager/0.log" Jul 24 23:07:12.942677 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:12.942647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-c45m2_bbad459a-837d-46ed-875b-c241c1d64072/kube-auth-proxy/0.log" Jul 24 23:07:12.973728 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:12.973689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/0.log" Jul 24 23:07:13.001917 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:13.001885 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/1.log" Jul 24 23:07:14.601059 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:14.601029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58b94969-c45m2_bbad459a-837d-46ed-875b-c241c1d64072/kube-auth-proxy/0.log" Jul 24 23:07:14.630106 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:14.630074 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/0.log" Jul 24 23:07:14.661652 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:14.661625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4xpv5_ab261fc0-d310-4080-9800-49de253bc260/istio-proxy/1.log" Jul 24 23:07:18.431916 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.431877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8f4qr_56481edd-b00d-48c6-88c7-b41cc816ad04/kube-multus/0.log" Jul 24 23:07:18.571625 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.571599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/kube-multus-additional-cni-plugins/0.log" Jul 24 23:07:18.588020 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.587997 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/egress-router-binary-copy/0.log" Jul 24 23:07:18.603619 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.603598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/cni-plugins/0.log" Jul 24 23:07:18.619042 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.619022 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/bond-cni-plugin/0.log" Jul 24 23:07:18.634239 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.634213 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/routeoverride-cni/0.log" Jul 24 23:07:18.649345 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.649328 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/whereabouts-cni-bincopy/0.log" Jul 24 23:07:18.665456 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.665437 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b8kmm_4689ab96-2a41-4e2e-8f6b-243f97b29cdc/whereabouts-cni/0.log" Jul 24 23:07:18.890209 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.890183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vhmh_f1654044-8118-4cc6-8743-397a4adab49f/network-metrics-daemon/0.log" Jul 24 23:07:18.903341 ip-10-0-132-132 kubenswrapper[2576]: I0724 23:07:18.903313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vhmh_f1654044-8118-4cc6-8743-397a4adab49f/kube-rbac-proxy/0.log"