Apr 24 16:39:19.122951 ip-10-0-139-51 systemd[1]: Starting Kubernetes Kubelet... Apr 24 16:39:19.611663 ip-10-0-139-51 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 24 16:39:19.611663 ip-10-0-139-51 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 24 16:39:19.611663 ip-10-0-139-51 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 24 16:39:19.611663 ip-10-0-139-51 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 24 16:39:19.611663 ip-10-0-139-51 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 24 16:39:19.614948 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.614861 2561 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 24 16:39:19.618038 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618023 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 24 16:39:19.618038 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618037 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618041 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618045 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618048 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618052 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618055 2561 feature_gate.go:328] unrecognized feature gate: Example2 Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618058 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618062 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618064 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618072 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618075 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618078 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618080 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618083 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618085 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618088 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618091 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618093 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618096 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618099 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 24 16:39:19.618094 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618101 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618104 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618107 2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618110 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618112 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618115 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618117 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618120 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618122 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618125 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618127 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618130 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618132 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618135 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618137 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618140 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618142 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618145 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618147 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618149 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 24 16:39:19.618580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618152 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618154 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618157 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618159 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618162 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618164 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618168 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618170 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618173 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618175 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618178 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618180 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618182 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618185 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618189 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618191 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618194 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618196 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618199 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618201 2561 feature_gate.go:328] unrecognized feature gate: Example Apr 24 16:39:19.619081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618204 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618206 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618209 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618213 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618217 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618221 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618224 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618226 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618229 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618231 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618234 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618236 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618239 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618241 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618243 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618247 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618250 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618252 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618254 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 24 16:39:19.619548 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618258 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618262 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618264 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618266 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618269 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.618271 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619975 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619982 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619985 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619989 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619992 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619995 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.619997 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620000 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620002 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620006 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620010 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620012 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620015 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 24 16:39:19.620039 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620017 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620019 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620022 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620026 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620029 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620032 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620034 2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620037 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620039 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620050 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620053 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620055 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620058 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620061 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620063 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620066 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620068 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620070 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620073 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 24 16:39:19.620541 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620075 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620078 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620080 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620082 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620085 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620087 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620089 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620092 2561 feature_gate.go:328] unrecognized feature gate: Example Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620094 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620097 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620099 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620101 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620104 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620106 2561 feature_gate.go:328] unrecognized feature gate: Example2 Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620108 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620112 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620115 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620117 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620120 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620123 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 24 16:39:19.621045 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620125 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620128 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620130 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620133 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620136 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620138 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620140 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620143 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620145 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620147 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620150 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620152 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620155 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620158 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620160 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620163 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620166 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620168 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620170 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 24 16:39:19.621524 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620173 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620175 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620177 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620180 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620182 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620184 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620187 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620189 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620193 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620196 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620198 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620201 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620203 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620205 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.620208 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621578 2561 flags.go:64] FLAG: --address="0.0.0.0" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621587 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621595 2561 flags.go:64] FLAG: --anonymous-auth="true" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621600 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621605 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621609 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 24 16:39:19.621982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621613 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621617 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621620 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621623 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621627 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621630 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621633 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621636 2561 flags.go:64] FLAG: --cgroup-root="" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621638 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621641 2561 flags.go:64] FLAG: --client-ca-file="" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621644 2561 flags.go:64] FLAG: --cloud-config="" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621647 2561 flags.go:64] FLAG: --cloud-provider="external" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621650 2561 flags.go:64] FLAG: --cluster-dns="[]" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621653 2561 flags.go:64] FLAG: --cluster-domain="" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621656 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621659 2561 flags.go:64] FLAG: --config-dir="" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621662 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621676 2561 flags.go:64] FLAG: --container-log-max-files="5" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621680 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621683 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621686 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621690 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621693 2561 flags.go:64] FLAG: --contention-profiling="false" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621696 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 24 16:39:19.622482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621699 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621702 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621704 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621708 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621711 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621721 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621724 2561 flags.go:64] FLAG: --enable-load-reader="false" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621727 2561 flags.go:64] FLAG: --enable-server="true" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621729 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621734 2561 flags.go:64] FLAG: --event-burst="100" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621737 2561 flags.go:64] FLAG: --event-qps="50" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621740 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621743 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621746 2561 flags.go:64] FLAG: --eviction-hard="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621750 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621752 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621755 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621758 2561 flags.go:64] FLAG: --eviction-soft="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621761 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621764 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621767 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621770 2561 flags.go:64] FLAG: --experimental-mounter-path="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621773 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621776 2561 flags.go:64] FLAG: --fail-swap-on="true" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621778 2561 flags.go:64] FLAG: --feature-gates="" Apr 24 16:39:19.623054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621782 2561 flags.go:64] FLAG: --file-check-frequency="20s" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621785 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621788 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621791 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621794 2561 flags.go:64] FLAG: --healthz-port="10248" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621797 2561 flags.go:64] FLAG: --help="false" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621800 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621803 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621806 2561 flags.go:64] FLAG: --http-check-frequency="20s" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621809 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621812 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621826 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621831 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621836 2561 flags.go:64] FLAG: --image-service-endpoint="" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621841 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621844 2561 flags.go:64] FLAG: --kube-api-burst="100" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621847 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621850 2561 flags.go:64] FLAG: --kube-api-qps="50" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621853 2561 flags.go:64] FLAG: --kube-reserved="" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621856 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621859 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621862 2561 flags.go:64] FLAG: --kubelet-cgroups="" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621865 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621868 2561 flags.go:64] FLAG: --lock-file="" Apr 24 16:39:19.623620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621871 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621874 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621877 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621882 2561 flags.go:64] FLAG: --log-json-split-stream="false" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621885 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621888 2561 flags.go:64] FLAG: --log-text-split-stream="false" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621891 2561 flags.go:64] FLAG: --logging-format="text" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621894 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621897 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621900 2561 flags.go:64] FLAG: --manifest-url="" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621903 2561 flags.go:64] FLAG: --manifest-url-header="" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621907 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621910 2561 flags.go:64] FLAG: --max-open-files="1000000" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621914 2561 flags.go:64] FLAG: --max-pods="110" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621917 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621920 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621923 2561 flags.go:64] FLAG: --memory-manager-policy="None" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621926 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621929 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621932 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621934 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621942 2561 flags.go:64] FLAG: --node-status-max-images="50" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621945 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621948 2561 flags.go:64] FLAG: --oom-score-adj="-999" Apr 24 16:39:19.624192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621951 2561 flags.go:64] FLAG: --pod-cidr="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621954 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621959 2561 flags.go:64] FLAG: --pod-manifest-path="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621962 2561 flags.go:64] FLAG: --pod-max-pids="-1" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621965 2561 flags.go:64] FLAG: --pods-per-core="0" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621968 2561 flags.go:64] FLAG: --port="10250" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621971 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621974 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c03db12054f03a47" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621977 2561 flags.go:64] FLAG: --qos-reserved="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621980 2561 flags.go:64] FLAG: --read-only-port="10255" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621983 2561 flags.go:64] FLAG: --register-node="true" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621986 2561 flags.go:64] FLAG: --register-schedulable="true" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621989 2561 flags.go:64] FLAG: --register-with-taints="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621993 2561 flags.go:64] FLAG: --registry-burst="10" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621996 2561 flags.go:64] FLAG: --registry-qps="5" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.621999 2561 flags.go:64] FLAG: --reserved-cpus="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622002 2561 flags.go:64] FLAG: --reserved-memory="" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622005 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622008 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622011 2561 flags.go:64] FLAG: --rotate-certificates="false" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622014 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622017 2561 flags.go:64] FLAG: --runonce="false" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622020 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622022 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622025 2561 flags.go:64] FLAG: --seccomp-default="false" Apr 24 16:39:19.624765 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622028 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622031 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622034 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622036 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622040 2561 flags.go:64] FLAG: --storage-driver-password="root" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622042 2561 flags.go:64] FLAG: --storage-driver-secure="false" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622045 2561 flags.go:64] FLAG: --storage-driver-table="stats" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622048 2561 flags.go:64] FLAG: --storage-driver-user="root" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622053 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622056 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622058 2561 flags.go:64] FLAG: --system-cgroups="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622061 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622066 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622069 2561 flags.go:64] FLAG: --tls-cert-file="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622072 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622075 2561 flags.go:64] FLAG: --tls-min-version="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622078 2561 flags.go:64] FLAG: --tls-private-key-file="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622080 2561 flags.go:64] FLAG: --topology-manager-policy="none" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622083 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622086 2561 flags.go:64] FLAG: --topology-manager-scope="container" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622089 2561 flags.go:64] FLAG: --v="2" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622093 2561 flags.go:64] FLAG: --version="false" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622097 2561 flags.go:64] FLAG: --vmodule="" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622102 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.622105 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 24 16:39:19.625368 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622210 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622214 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622217 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622220 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622223 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622226 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622228 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622232 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622236 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622239 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622242 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622244 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622247 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622249 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622259 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622263 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622266 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622269 2561 feature_gate.go:328] unrecognized feature gate: Example Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622271 2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 24 16:39:19.625970 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622274 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622276 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622279 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622281 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622284 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622287 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622289 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622292 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622294 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622297 2561 feature_gate.go:328] unrecognized feature gate: Example2 Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622307 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622310 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622312 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622315 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622318 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622320 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622322 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622325 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622327 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622330 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 24 16:39:19.626461 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622332 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622335 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622338 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622341 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622343 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622345 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622348 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622350 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622360 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622363 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622365 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622367 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622370 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622373 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622376 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622378 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622381 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622383 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622385 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622388 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 24 16:39:19.626965 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622390 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622393 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622395 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622398 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622400 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622402 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622405 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622407 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622410 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622412 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622415 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622417 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622419 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622422 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622424 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622426 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622429 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622431 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622435 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 24 16:39:19.627427 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622438 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622442 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622446 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622449 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622451 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622454 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622459 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.622462 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 24 16:39:19.627882 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.623284 2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 24 16:39:19.632081 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.632058 2561 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 24 16:39:19.632081 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.632079 2561 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632141 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632148 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632153 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632157 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632164 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632171 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632175 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632179 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632183 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632187 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632191 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632195 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632199 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632202 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632206 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632210 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632214 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632218 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 24 16:39:19.632254 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632222 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632226 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632230 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632234 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632238 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632243 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632247 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632251 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632255 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632259 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632263 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632267 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632272 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632276 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632281 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632284 2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632288 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632292 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632296 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632300 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 24 16:39:19.633081 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632304 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632308 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632312 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632316 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632320 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632323 2561 feature_gate.go:328] unrecognized feature gate: Example2 Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632327 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632331 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632336 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632339 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632344 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632348 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632352 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632356 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632360 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632365 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632369 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632373 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632377 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632381 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 24 16:39:19.633616 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632385 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632389 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632393 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632397 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632401 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632405 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632409 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632413 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632417 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632422 2561 feature_gate.go:328] unrecognized feature gate: Example Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632425 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632429 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632433 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632437 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632442 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632446 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632450 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632454 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632459 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632463 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 24 16:39:19.634216 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632467 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632471 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632475 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632479 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632484 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632491 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632497 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632501 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.632509 2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632653 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632661 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632681 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632687 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632691 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 24 16:39:19.634990 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632695 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632699 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632705 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632712 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632716 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632721 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632726 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632730 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632734 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632739 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632743 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632747 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632750 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632755 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632759 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632764 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632768 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632774 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632778 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 24 16:39:19.635369 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632783 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632788 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632792 2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632796 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632800 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632804 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632808 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632812 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632816 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632820 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632824 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632828 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632832 2561 feature_gate.go:328] unrecognized feature gate: Example2 Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632836 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632839 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632844 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632848 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632852 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632856 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632860 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 24 16:39:19.635857 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632864 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632868 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632872 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632876 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632880 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632884 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632888 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632892 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632896 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632900 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632904 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632909 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632913 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632917 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632922 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632927 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632931 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632935 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632939 2561 feature_gate.go:328] unrecognized feature gate: Example Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632943 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 24 16:39:19.636328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632947 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632951 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632954 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632958 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632962 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632966 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632970 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632974 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632978 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632982 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632986 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632991 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632994 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.632998 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633002 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633006 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633010 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633014 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633018 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633022 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 24 16:39:19.636847 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633026 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 24 16:39:19.637322 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:19.633030 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 24 16:39:19.637322 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.633038 2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 24 16:39:19.637322 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.633879 2561 server.go:962] "Client rotation is on, will bootstrap in background" Apr 24 16:39:19.639961 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.639944 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 24 16:39:19.640868 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.640854 2561 server.go:1019] "Starting client certificate rotation" Apr 24 16:39:19.640970 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.640953 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 24 16:39:19.641005 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.640998 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 24 16:39:19.670576 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.670553 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 24 16:39:19.673499 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.673480 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 24 16:39:19.686405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.686386 2561 log.go:25] "Validated CRI v1 runtime API" Apr 24 16:39:19.694661 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.694638 2561 log.go:25] "Validated CRI v1 image API" Apr 24 16:39:19.696737 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.696722 2561 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 24 16:39:19.700840 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.700819 2561 fs.go:135] Filesystem UUIDs: map[01a28bbf-5e5f-4bad-8a56-d8dcceb3628c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ffcc013d-3240-4d87-8cad-b3494eec3132:/dev/nvme0n1p4] Apr 24 16:39:19.700912 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.700839 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 24 16:39:19.704245 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.704221 2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 24 16:39:19.706677 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.706553 2561 manager.go:217] Machine: {Timestamp:2026-04-24 16:39:19.704441149 +0000 UTC m=+0.453611901 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3203972 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a8fc5bd2812e87dd872c7943308ed SystemUUID:ec2a8fc5-bd28-12e8-7dd8-72c7943308ed BootID:cdeeabc2-4535-4bd2-bcc8-7a57ceb8a3cd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:31:22:1b:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:31:22:1b:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:db:84:34:e1:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 24 16:39:19.706677 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.706652 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 24 16:39:19.706829 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.706810 2561 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 24 16:39:19.709900 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.709877 2561 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 24 16:39:19.710063 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.709902 2561 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 24 16:39:19.710147 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.710077 2561 topology_manager.go:138] "Creating topology manager with none policy" Apr 24 16:39:19.710147 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.710089 2561 container_manager_linux.go:306] "Creating device plugin manager" Apr 24 16:39:19.710147 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.710107 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 24 16:39:19.710147 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.710124 2561 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 24 16:39:19.711100 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.711087 2561 state_mem.go:36] "Initialized new in-memory state store" Apr 24 16:39:19.711372 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.711360 2561 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 24 16:39:19.716090 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.716078 2561 kubelet.go:491] "Attempting to sync node with API server" Apr 24 16:39:19.716159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.716095 2561 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 24 16:39:19.716159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.716111 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 24 16:39:19.716159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.716122 2561 kubelet.go:397] "Adding apiserver pod source" Apr 24 16:39:19.716159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.716134 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 24 16:39:19.717892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.717879 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 24 16:39:19.717953 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.717901 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 24 16:39:19.721893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.721876 2561 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 24 16:39:19.723368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.723337 2561 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 24 16:39:19.725318 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725301 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725321 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725328 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725334 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725340 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725346 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725360 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725366 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725372 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 24 16:39:19.725380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725379 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 24 16:39:19.725617 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725391 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 24 16:39:19.725617 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.725400 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 24 16:39:19.726375 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.726366 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 24 16:39:19.726375 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.726376 2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 24 16:39:19.728607 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.728585 2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Apr 24 16:39:19.729039 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.729018 2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Apr 24 16:39:19.729641 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.729623 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9ccw" Apr 24 16:39:19.730347 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.730335 2561 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 24 16:39:19.730397 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.730388 2561 server.go:1295] "Started kubelet" Apr 24 16:39:19.730525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.730490 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 24 16:39:19.730594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.730535 2561 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 24 16:39:19.730699 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.730617 2561 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 24 16:39:19.731241 ip-10-0-139-51 systemd[1]: Started Kubernetes Kubelet. Apr 24 16:39:19.731894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.731876 2561 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 24 16:39:19.732545 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.732534 2561 server.go:317] "Adding debug handlers to kubelet server" Apr 24 16:39:19.737621 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.737451 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9ccw" Apr 24 16:39:19.737984 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.737967 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 24 16:39:19.738559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.738544 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 24 16:39:19.739885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.739870 2561 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 24 16:39:19.739982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.739973 2561 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 24 16:39:19.744102 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.739659 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-51.ec2.internal.18a9586e88469c4f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-51.ec2.internal,UID:ip-10-0-139-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-51.ec2.internal,},FirstTimestamp:2026-04-24 16:39:19.730347087 +0000 UTC m=+0.479517839,LastTimestamp:2026-04-24 16:39:19.730347087 +0000 UTC m=+0.479517839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-51.ec2.internal,}" Apr 24 16:39:19.744578 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.739872 2561 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Apr 24 16:39:19.744722 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.744706 2561 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 24 16:39:19.744924 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.744900 2561 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Apr 24 16:39:19.745129 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745112 2561 factory.go:55] Registering systemd factory Apr 24 16:39:19.745204 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745134 2561 factory.go:223] Registration of the systemd container factory successfully Apr 24 16:39:19.745586 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.745566 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:19.745703 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745688 2561 reconstruct.go:97] "Volume reconstruction finished" Apr 24 16:39:19.745703 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745703 2561 reconciler.go:26] "Reconciler: start to sync state" Apr 24 16:39:19.745872 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745857 2561 factory.go:153] Registering CRI-O factory Apr 24 16:39:19.745872 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745873 2561 factory.go:223] Registration of the crio container factory successfully Apr 24 16:39:19.746011 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.745984 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 24 16:39:19.746011 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.746008 2561 factory.go:103] Registering Raw factory Apr 24 16:39:19.746092 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.746024 2561 manager.go:1196] Started watching for new ooms in manager Apr 24 16:39:19.746574 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.746559 2561 manager.go:319] Starting recovery of all containers Apr 24 16:39:19.752280 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.752261 2561 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:19.755347 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.755326 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.755988 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.755973 2561 manager.go:324] Recovery completed Apr 24 16:39:19.759575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.759563 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.762233 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762219 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.762297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762260 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.762297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762275 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.762736 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762723 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 24 16:39:19.762736 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762734 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 24 16:39:19.762818 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.762767 2561 state_mem.go:36] "Initialized new in-memory state store" Apr 24 16:39:19.765351 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.765339 2561 policy_none.go:49] "None policy: Start" Apr 24 16:39:19.765402 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.765355 2561 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 24 16:39:19.765402 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.765364 2561 state_mem.go:35] "Initializing new in-memory state store" Apr 24 16:39:19.816201 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816188 2561 manager.go:341] "Starting Device Plugin manager" Apr 24 16:39:19.816304 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.816215 2561 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 24 16:39:19.816304 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816224 2561 server.go:85] "Starting device plugin registration server" Apr 24 16:39:19.816442 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816424 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 24 16:39:19.816531 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816440 2561 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 24 16:39:19.816606 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816586 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 24 16:39:19.816718 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816705 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 24 16:39:19.816718 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.816718 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 24 16:39:19.817112 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.817093 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 24 16:39:19.817185 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.817131 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:19.843942 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.843910 2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 24 16:39:19.845054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.845037 2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 24 16:39:19.845140 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.845062 2561 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 24 16:39:19.845140 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.845093 2561 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 24 16:39:19.845140 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.845099 2561 kubelet.go:2451] "Starting kubelet main sync loop" Apr 24 16:39:19.845140 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.845140 2561 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 24 16:39:19.847564 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.847544 2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:19.916593 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.916544 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.917381 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.917367 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.917446 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.917397 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.917446 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.917411 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.917446 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.917434 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.925778 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.925763 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.925831 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.925785 2561 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-51.ec2.internal\": node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:19.945022 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.945004 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:19.946122 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.946107 2561 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"] Apr 24 16:39:19.946172 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.946164 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.947470 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.947456 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.947523 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.947485 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.947523 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.947494 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.948744 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.948732 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.948918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.948906 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.948955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.948931 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.949420 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949407 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.949483 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949429 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.949483 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949442 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.949555 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949410 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.949555 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949519 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.949555 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.949541 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.950836 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.950824 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.950881 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.950845 2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 24 16:39:19.951455 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.951440 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory" Apr 24 16:39:19.951516 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.951464 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure" Apr 24 16:39:19.951516 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:19.951473 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID" Apr 24 16:39:19.983181 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.983157 2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal" Apr 24 16:39:19.987235 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:19.987219 2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.045263 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.045247 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.046420 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.046405 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/964185a562d5ec022f05355b7dc81013-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"964185a562d5ec022f05355b7dc81013\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.046472 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.046431 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.046472 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.046447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.145736 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.145702 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.146854 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146839 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.146897 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146865 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.146897 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146886 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/964185a562d5ec022f05355b7dc81013-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"964185a562d5ec022f05355b7dc81013\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.146956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146936 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/964185a562d5ec022f05355b7dc81013-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"964185a562d5ec022f05355b7dc81013\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.146956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146944 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.147015 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.146936 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41c83e021f241dcae39ca57bc9bf5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"41c83e021f241dcae39ca57bc9bf5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.246334 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.246289 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.284528 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.284514 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.290063 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.290043 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:20.346961 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.346937 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.447346 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.447324 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.547887 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.547823 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.641180 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.641157 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 24 16:39:20.641799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.641283 2561 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 24 16:39:20.641799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.641317 2561 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 24 16:39:20.648469 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.648453 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.739108 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.739079 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 24 16:39:20.744463 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.744427 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-23 16:34:19 +0000 UTC" deadline="2027-12-01 11:52:34.503169593 +0000 UTC" Apr 24 16:39:20.744463 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.744458 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14059h13m13.758714948s" Apr 24 16:39:20.749185 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.749169 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.761624 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.761609 2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 24 16:39:20.771065 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:20.771041 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964185a562d5ec022f05355b7dc81013.slice/crio-bd129c0bc3caeee787ff2124237e1623dfcd58ad9978a52a1f3b90e9b4fd8b8e WatchSource:0}: Error finding container bd129c0bc3caeee787ff2124237e1623dfcd58ad9978a52a1f3b90e9b4fd8b8e: Status 404 returned error can't find the container with id bd129c0bc3caeee787ff2124237e1623dfcd58ad9978a52a1f3b90e9b4fd8b8e Apr 24 16:39:20.771475 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:20.771456 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c83e021f241dcae39ca57bc9bf5fb2.slice/crio-2934db6b8ef5c05e08953a88317aaeac88464f210b9feb827a75fa6d244fb05b WatchSource:0}: Error finding container 2934db6b8ef5c05e08953a88317aaeac88464f210b9feb827a75fa6d244fb05b: Status 404 returned error can't find the container with id 2934db6b8ef5c05e08953a88317aaeac88464f210b9feb827a75fa6d244fb05b Apr 24 16:39:20.775456 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.775439 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 16:39:20.786553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.786535 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ll998" Apr 24 16:39:20.794376 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.794360 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ll998" Apr 24 16:39:20.847393 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.847323 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"964185a562d5ec022f05355b7dc81013","Type":"ContainerStarted","Data":"bd129c0bc3caeee787ff2124237e1623dfcd58ad9978a52a1f3b90e9b4fd8b8e"} Apr 24 16:39:20.848288 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:20.848268 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"41c83e021f241dcae39ca57bc9bf5fb2","Type":"ContainerStarted","Data":"2934db6b8ef5c05e08953a88317aaeac88464f210b9feb827a75fa6d244fb05b"} Apr 24 16:39:20.849372 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.849358 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:20.949858 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:20.949838 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:21.050256 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.050231 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:21.083630 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.083606 2561 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:21.150854 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.150782 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:21.251490 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.251454 2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found" Apr 24 16:39:21.325210 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.325047 2561 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:21.345441 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.345238 2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" Apr 24 16:39:21.357044 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.356940 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 24 16:39:21.357944 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.357925 2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" Apr 24 16:39:21.365826 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.365808 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 24 16:39:21.717206 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.717177 2561 apiserver.go:52] "Watching apiserver" Apr 24 16:39:21.724148 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.724123 2561 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 24 16:39:21.725605 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.725578 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mwzw9","openshift-network-diagnostics/network-check-target-mhwg5","openshift-cluster-node-tuning-operator/tuned-7vwnx","openshift-dns/node-resolver-zrlj6","openshift-multus/multus-q6scl","openshift-network-operator/iptables-alerter-gx57q","openshift-ovn-kubernetes/ovnkube-node-xxfbv","kube-system/konnectivity-agent-srzmm","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7","openshift-image-registry/node-ca-c4586","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","openshift-multus/multus-additional-cni-plugins-qkvcg"] Apr 24 16:39:21.727111 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.727085 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.728217 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.728196 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.729223 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.729205 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.729538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.729515 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 24 16:39:21.729812 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.729781 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5ml2p\"" Apr 24 16:39:21.729895 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.729827 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 24 16:39:21.730144 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730095 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.730417 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730392 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 24 16:39:21.730560 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730543 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.730626 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730578 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 24 16:39:21.730626 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730604 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.730988 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730960 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rg4d4\"" Apr 24 16:39:21.730988 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.730971 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 24 16:39:21.731425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.731406 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jbtpc\"" Apr 24 16:39:21.731523 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.731505 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.731622 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.731598 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.733132 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.733114 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.733223 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.733208 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 24 16:39:21.734099 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.734078 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-phdpd\"" Apr 24 16:39:21.734099 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.734091 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.734445 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.734418 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.734898 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.734877 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.735111 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.735088 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.736114 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.736086 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j8bfh\"" Apr 24 16:39:21.736114 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.736100 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 24 16:39:21.736959 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.736940 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.737044 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.736959 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 24 16:39:21.737580 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.737562 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.738192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.738175 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.738300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.738284 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.738362 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.738341 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qvpdb\"" Apr 24 16:39:21.738435 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.738401 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-42nv5\"" Apr 24 16:39:21.738559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.738541 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.739067 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.739051 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.740319 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740303 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9mjfw\"" Apr 24 16:39:21.740441 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740304 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 24 16:39:21.740441 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740305 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 24 16:39:21.740627 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740609 2561 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:21.740894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740748 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.740894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.740894 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 24 16:39:21.741240 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.741223 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.741875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.741856 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 24 16:39:21.741974 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.741895 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 24 16:39:21.741974 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.741917 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 24 16:39:21.741974 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.741962 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-flcs7\"" Apr 24 16:39:21.742291 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.742263 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:21.742382 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.742334 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:21.742382 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.742280 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.742485 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.742427 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:21.742771 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.742757 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 24 16:39:21.746008 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.745992 2561 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 24 16:39:21.748679 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.748646 2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 24 16:39:21.754172 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754150 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-modprobe-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.754262 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754186 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-slash\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754262 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754240 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-etc-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754376 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-system-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.754376 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-konnectivity-ca\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.754472 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754396 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-conf\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.754472 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-kubelet\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754472 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-socket-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.754623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754471 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.754623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754519 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.754623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754555 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wsc\" (UniqueName: \"kubernetes.io/projected/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-kube-api-access-c7wsc\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.754623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754579 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-log-socket\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-host\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754625 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-script-lib\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754652 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-system-cni-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t67\" (UniqueName: \"kubernetes.io/projected/b21bb7b3-2bb8-4576-b27b-a786cd88c140-kube-api-access-x6t67\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754719 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-systemd-units\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754749 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754772 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnsx\" (UniqueName: \"kubernetes.io/projected/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-kube-api-access-xlnsx\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754791 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-run\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cstt\" (UniqueName: \"kubernetes.io/projected/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-kube-api-access-4cstt\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.754875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-k8s-cni-cncf-io\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-hostroot\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754912 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-serviceca\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-netd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc972117-5b3e-4446-8204-6290e86329ad-ovn-node-metrics-cert\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.754984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-conf-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-ovn\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-node-log\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755040 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-bin\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-env-overrides\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755116 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-cni-binary-copy\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-tuned\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.755199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cnibin\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755218 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-netns\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755250 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-var-lib-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-host-slash\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-kubernetes\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-sys\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755394 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fch7b\" (UniqueName: \"kubernetes.io/projected/dc972117-5b3e-4446-8204-6290e86329ad-kube-api-access-fch7b\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rdk\" (UniqueName: \"kubernetes.io/projected/da21c390-2975-4210-a740-ee0091b9c5b0-kube-api-access-f7rdk\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfcb\" (UniqueName: \"kubernetes.io/projected/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kube-api-access-qjfcb\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755473 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvr7\" (UniqueName: \"kubernetes.io/projected/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-kube-api-access-jwvr7\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755494 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-systemd\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-var-lib-kubelet\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.755553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-tmp\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-tmp-dir\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755581 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-os-release\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-socket-dir-parent\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755642 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-bin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-etc-kubernetes\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-agent-certs\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-registration-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-device-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755791 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6xx\" (UniqueName: \"kubernetes.io/projected/b71bdf81-e127-49a6-aca6-ff4846028fd9-kube-api-access-sq6xx\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-os-release\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755847 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-hosts-file\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755891 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-cnibin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-multus-daemon-config\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755934 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-multus-certs\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysconfig\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.755989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-systemd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-multus\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-iptables-alerter-script\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756154 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-lib-modules\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756230 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-host\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756255 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-config\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-netns\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-sys-fs\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756381 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.756784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.756425 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-kubelet\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.795136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.795110 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 16:34:20 +0000 UTC" deadline="2027-12-31 17:55:58.151104218 +0000 UTC" Apr 24 16:39:21.795136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.795134 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14785h16m36.35597314s" Apr 24 16:39:21.857191 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857160 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-systemd\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.857342 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857238 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-systemd\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.857342 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857273 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-var-lib-kubelet\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.857342 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857311 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-tmp\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.857342 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857333 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-tmp-dir\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857356 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-os-release\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857397 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-var-lib-kubelet\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857430 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-os-release\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857465 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-socket-dir-parent\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857490 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-bin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857513 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-etc-kubernetes\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857552 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857539 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-agent-certs\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857562 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-registration-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857572 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-socket-dir-parent\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857589 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-device-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857631 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6xx\" (UniqueName: \"kubernetes.io/projected/b71bdf81-e127-49a6-aca6-ff4846028fd9-kube-api-access-sq6xx\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857637 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-device-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857651 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-bin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857692 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-tmp-dir\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857692 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-etc-kubernetes\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857680 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-os-release\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857692 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857745 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-registration-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857755 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857794 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-hosts-file\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857827 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-cnibin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857848 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-multus-daemon-config\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857870 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-multus-certs\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.857885 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857894 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysconfig\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857917 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-systemd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857944 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857967 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857963 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-os-release\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.857990 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-multus\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858044 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysconfig\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858063 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-hosts-file\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858068 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-iptables-alerter-script\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858098 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-cnibin\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858122 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858143 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858150 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-lib-modules\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858189 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-host\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858218 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-config\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858243 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-netns\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858268 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.858551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858291 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858300 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-sys-fs\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858339 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-multus-certs\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858342 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858360 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858371 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858376 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858398 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-kubelet\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858423 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-modprobe-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858427 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-netns\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858449 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-slash\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858456 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-lib-modules\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858464 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-systemd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858463 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-multus-daemon-config\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858477 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-etc-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858508 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-cni-multus\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858518 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-host\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.859315 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858520 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-iptables-alerter-script\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858519 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-etc-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858548 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-slash\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858556 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-system-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858561 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858584 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-var-lib-kubelet\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858583 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-konnectivity-ca\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858615 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-conf\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858627 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-system-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858584 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.858657 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858695 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-kubelet\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858640 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-kubelet\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.858758 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:22.358726837 +0000 UTC m=+3.107897579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858763 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-sysctl-conf\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858777 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-socket-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858799 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-modprobe-d\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.860046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858805 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858847 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858873 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wsc\" (UniqueName: \"kubernetes.io/projected/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-kube-api-access-c7wsc\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858912 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-socket-dir\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858902 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-log-socket\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858944 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-host\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858968 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-script-lib\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858993 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-system-cni-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.858997 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-host\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859017 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t67\" (UniqueName: \"kubernetes.io/projected/b21bb7b3-2bb8-4576-b27b-a786cd88c140-kube-api-access-x6t67\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859042 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-systemd-units\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859057 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-config\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859067 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-konnectivity-ca\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859070 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859107 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnsx\" (UniqueName: \"kubernetes.io/projected/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-kube-api-access-xlnsx\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859128 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-sys-fs\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859132 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-run\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.860801 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859156 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4cstt\" (UniqueName: \"kubernetes.io/projected/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-kube-api-access-4cstt\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859162 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-system-cni-dir\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859181 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-k8s-cni-cncf-io\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859205 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-hostroot\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859229 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-serviceca\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859251 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-netd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859276 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc972117-5b3e-4446-8204-6290e86329ad-ovn-node-metrics-cert\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859327 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-host-run-k8s-cni-cncf-io\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859337 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859352 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-conf-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859379 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-ovn\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859383 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/872ed659-a1a1-468f-8ab1-097f13c7e6eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859398 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-hostroot\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859435 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-run-ovn\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859440 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-node-log\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859455 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-ovnkube-script-lib\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859403 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-node-log\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859510 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-conf-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.861538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859479 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-run\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859542 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-bin\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859554 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-log-socket\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859510 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-bin\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859573 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859592 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-env-overrides\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859578 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-systemd-units\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859626 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-cni-netd\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859694 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859749 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-cni-binary-copy\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859778 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-tuned\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.859959 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-serviceca\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860200 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cnibin\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860232 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-netns\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860258 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-var-lib-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860292 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-host-slash\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860320 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:21.862409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860345 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860379 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-kubernetes\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860405 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-sys\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860430 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fch7b\" (UniqueName: \"kubernetes.io/projected/dc972117-5b3e-4446-8204-6290e86329ad-kube-api-access-fch7b\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860454 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rdk\" (UniqueName: \"kubernetes.io/projected/da21c390-2975-4210-a740-ee0091b9c5b0-kube-api-access-f7rdk\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860460 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da21c390-2975-4210-a740-ee0091b9c5b0-cni-binary-copy\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860479 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfcb\" (UniqueName: \"kubernetes.io/projected/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kube-api-access-qjfcb\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860506 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvr7\" (UniqueName: \"kubernetes.io/projected/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-kube-api-access-jwvr7\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860625 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc972117-5b3e-4446-8204-6290e86329ad-env-overrides\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860709 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-var-lib-openvswitch\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860763 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-host-slash\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860785 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc972117-5b3e-4446-8204-6290e86329ad-host-run-netns\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860809 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-sys\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860891 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da21c390-2975-4210-a740-ee0091b9c5b0-multus-cni-dir\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.860933 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cnibin\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.861111 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-kubernetes\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863165 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.861333 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b21bb7b3-2bb8-4576-b27b-a786cd88c140-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.863918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.863389 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-etc-tuned\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.863402 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-tmp\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.863918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.863761 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc972117-5b3e-4446-8204-6290e86329ad-ovn-node-metrics-cert\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.863918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.863828 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/813b1e82-26e4-4a64-a2d6-d4bc774b92e2-agent-certs\") pod \"konnectivity-agent-srzmm\" (UID: \"813b1e82-26e4-4a64-a2d6-d4bc774b92e2\") " pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:21.866597 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.866567 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:21.866597 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.866598 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:21.866787 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.866608 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:21.866787 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:21.866685 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:22.366655661 +0000 UTC m=+3.115826400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:21.867367 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.867348 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6xx\" (UniqueName: \"kubernetes.io/projected/b71bdf81-e127-49a6-aca6-ff4846028fd9-kube-api-access-sq6xx\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:21.868516 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.868491 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wsc\" (UniqueName: \"kubernetes.io/projected/2d1c3528-cf11-4ef1-bb88-2f59df7e45a9-kube-api-access-c7wsc\") pod \"node-ca-c4586\" (UID: \"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9\") " pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:21.868930 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.868911 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvr7\" (UniqueName: \"kubernetes.io/projected/9c0d56c6-7fdc-4b45-ba58-6da3aaccac01-kube-api-access-jwvr7\") pod \"iptables-alerter-gx57q\" (UID: \"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01\") " pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:21.869171 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.869137 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cstt\" (UniqueName: \"kubernetes.io/projected/afd8262e-c6d9-4d7a-89c6-4b87b7281a67-kube-api-access-4cstt\") pod \"node-resolver-zrlj6\" (UID: \"afd8262e-c6d9-4d7a-89c6-4b87b7281a67\") " pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:21.869295 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.869259 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnsx\" (UniqueName: \"kubernetes.io/projected/9bf0ebbf-8c09-4abd-b116-e82d4d4f8171-kube-api-access-xlnsx\") pod \"tuned-7vwnx\" (UID: \"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171\") " pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:21.870962 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.870939 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t67\" (UniqueName: \"kubernetes.io/projected/b21bb7b3-2bb8-4576-b27b-a786cd88c140-kube-api-access-x6t67\") pod \"multus-additional-cni-plugins-qkvcg\" (UID: \"b21bb7b3-2bb8-4576-b27b-a786cd88c140\") " pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:21.871795 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.871771 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rdk\" (UniqueName: \"kubernetes.io/projected/da21c390-2975-4210-a740-ee0091b9c5b0-kube-api-access-f7rdk\") pod \"multus-q6scl\" (UID: \"da21c390-2975-4210-a740-ee0091b9c5b0\") " pod="openshift-multus/multus-q6scl" Apr 24 16:39:21.872021 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.872005 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fch7b\" (UniqueName: \"kubernetes.io/projected/dc972117-5b3e-4446-8204-6290e86329ad-kube-api-access-fch7b\") pod \"ovnkube-node-xxfbv\" (UID: \"dc972117-5b3e-4446-8204-6290e86329ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:21.872239 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:21.872216 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfcb\" (UniqueName: \"kubernetes.io/projected/872ed659-a1a1-468f-8ab1-097f13c7e6eb-kube-api-access-qjfcb\") pod \"aws-ebs-csi-driver-node-6rgr7\" (UID: \"872ed659-a1a1-468f-8ab1-097f13c7e6eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:22.040942 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.040867 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:22.047572 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.047545 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" Apr 24 16:39:22.055247 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.055230 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" Apr 24 16:39:22.060821 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.060791 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4586" Apr 24 16:39:22.067037 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.067018 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q6scl" Apr 24 16:39:22.073562 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.073540 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" Apr 24 16:39:22.080079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.080058 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrlj6" Apr 24 16:39:22.085705 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.085689 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:22.091244 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.091227 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gx57q" Apr 24 16:39:22.324849 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.324813 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda21c390_2975_4210_a740_ee0091b9c5b0.slice/crio-819bf69eb696d3ef6bb707281787abb021270b1bb36cec56e3e31d92a93aa02c WatchSource:0}: Error finding container 819bf69eb696d3ef6bb707281787abb021270b1bb36cec56e3e31d92a93aa02c: Status 404 returned error can't find the container with id 819bf69eb696d3ef6bb707281787abb021270b1bb36cec56e3e31d92a93aa02c Apr 24 16:39:22.327525 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.327490 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd8262e_c6d9_4d7a_89c6_4b87b7281a67.slice/crio-063ce8163a4bd2b13fa5ff181a781495eb3a2df9b2dfb59823e750ce4595afe5 WatchSource:0}: Error finding container 063ce8163a4bd2b13fa5ff181a781495eb3a2df9b2dfb59823e750ce4595afe5: Status 404 returned error can't find the container with id 063ce8163a4bd2b13fa5ff181a781495eb3a2df9b2dfb59823e750ce4595afe5 Apr 24 16:39:22.330322 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.330299 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0d56c6_7fdc_4b45_ba58_6da3aaccac01.slice/crio-e9694bc40cc0a6385756353ada24db62587c81f929ac96a48526f0440decc8c2 WatchSource:0}: Error finding container e9694bc40cc0a6385756353ada24db62587c81f929ac96a48526f0440decc8c2: Status 404 returned error can't find the container with id e9694bc40cc0a6385756353ada24db62587c81f929ac96a48526f0440decc8c2 Apr 24 16:39:22.331194 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.331170 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21bb7b3_2bb8_4576_b27b_a786cd88c140.slice/crio-db2bd82d429c1d17c467183174bd996efac3fd2584896097dec1853081407110 WatchSource:0}: Error finding container db2bd82d429c1d17c467183174bd996efac3fd2584896097dec1853081407110: Status 404 returned error can't find the container with id db2bd82d429c1d17c467183174bd996efac3fd2584896097dec1853081407110 Apr 24 16:39:22.332262 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.332231 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1c3528_cf11_4ef1_bb88_2f59df7e45a9.slice/crio-6f9534c2630827503467852d11da9ff0b1436442f4979f6ffdc5fc5ca2852956 WatchSource:0}: Error finding container 6f9534c2630827503467852d11da9ff0b1436442f4979f6ffdc5fc5ca2852956: Status 404 returned error can't find the container with id 6f9534c2630827503467852d11da9ff0b1436442f4979f6ffdc5fc5ca2852956 Apr 24 16:39:22.333720 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.333696 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc972117_5b3e_4446_8204_6290e86329ad.slice/crio-45553c20c0273b4902ec37df3bd374f4397c2c0e87f91310d15c20864d87bffb WatchSource:0}: Error finding container 45553c20c0273b4902ec37df3bd374f4397c2c0e87f91310d15c20864d87bffb: Status 404 returned error can't find the container with id 45553c20c0273b4902ec37df3bd374f4397c2c0e87f91310d15c20864d87bffb Apr 24 16:39:22.333975 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:39:22.333948 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872ed659_a1a1_468f_8ab1_097f13c7e6eb.slice/crio-59238f965ab0af06fa96fc88c9fdc5cf906b7942e38df557937f06f36d29cdea WatchSource:0}: Error finding container 59238f965ab0af06fa96fc88c9fdc5cf906b7942e38df557937f06f36d29cdea: Status 404 returned error can't find the container with id 59238f965ab0af06fa96fc88c9fdc5cf906b7942e38df557937f06f36d29cdea Apr 24 16:39:22.364177 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.364041 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:22.364254 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.364155 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:22.364317 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.364307 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:23.364291033 +0000 UTC m=+4.113461783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:22.465521 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.465493 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:22.465683 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.465647 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:22.465739 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.465683 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:22.465739 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.465698 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:22.465804 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:22.465753 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:23.46573743 +0000 UTC m=+4.214908174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:22.795583 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.795446 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 16:34:20 +0000 UTC" deadline="2027-10-02 18:29:14.818443133 +0000 UTC" Apr 24 16:39:22.795583 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.795481 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12625h49m52.022965303s" Apr 24 16:39:22.859308 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.859246 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srzmm" event={"ID":"813b1e82-26e4-4a64-a2d6-d4bc774b92e2","Type":"ContainerStarted","Data":"d1d771c267a70f6345ed77cd5f0278f202a40821a96bfa821df1856d61035244"} Apr 24 16:39:22.868462 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.868332 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" event={"ID":"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171","Type":"ContainerStarted","Data":"7529007c3f8b9a5bee44ec7e03868fca6e13f0b36d754b66c00e9e573c614863"} Apr 24 16:39:22.876435 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.876389 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"45553c20c0273b4902ec37df3bd374f4397c2c0e87f91310d15c20864d87bffb"} Apr 24 16:39:22.891074 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.891043 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4586" event={"ID":"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9","Type":"ContainerStarted","Data":"6f9534c2630827503467852d11da9ff0b1436442f4979f6ffdc5fc5ca2852956"} Apr 24 16:39:22.896996 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.896970 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerStarted","Data":"db2bd82d429c1d17c467183174bd996efac3fd2584896097dec1853081407110"} Apr 24 16:39:22.903136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.903108 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gx57q" event={"ID":"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01","Type":"ContainerStarted","Data":"e9694bc40cc0a6385756353ada24db62587c81f929ac96a48526f0440decc8c2"} Apr 24 16:39:22.907486 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.907403 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" event={"ID":"872ed659-a1a1-468f-8ab1-097f13c7e6eb","Type":"ContainerStarted","Data":"59238f965ab0af06fa96fc88c9fdc5cf906b7942e38df557937f06f36d29cdea"} Apr 24 16:39:22.911888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.911864 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrlj6" event={"ID":"afd8262e-c6d9-4d7a-89c6-4b87b7281a67","Type":"ContainerStarted","Data":"063ce8163a4bd2b13fa5ff181a781495eb3a2df9b2dfb59823e750ce4595afe5"} Apr 24 16:39:22.913836 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.913793 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q6scl" event={"ID":"da21c390-2975-4210-a740-ee0091b9c5b0","Type":"ContainerStarted","Data":"819bf69eb696d3ef6bb707281787abb021270b1bb36cec56e3e31d92a93aa02c"} Apr 24 16:39:22.917004 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:22.916979 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"964185a562d5ec022f05355b7dc81013","Type":"ContainerStarted","Data":"e3210037aa1dd22f3b671535b8929733dfaad1e01cabeb13465ab907b8fc608a"} Apr 24 16:39:23.372787 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.372756 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:23.372971 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.372900 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:23.372971 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.372960 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:25.37294197 +0000 UTC m=+6.122112716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:23.475721 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.473884 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:23.475721 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.474075 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:23.475721 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.474093 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:23.475721 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.474106 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:23.475721 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.474168 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:25.4741484 +0000 UTC m=+6.223319153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:23.845815 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.845729 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:23.846239 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.845915 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:23.846316 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.845729 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:23.846421 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:23.846396 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:23.924696 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.924149 2561 generic.go:358] "Generic (PLEG): container finished" podID="41c83e021f241dcae39ca57bc9bf5fb2" containerID="640e916c1336d4a0432308197a63e477e5aaeae6131d24cab70b06812854243f" exitCode=0 Apr 24 16:39:23.924992 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.924877 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"41c83e021f241dcae39ca57bc9bf5fb2","Type":"ContainerDied","Data":"640e916c1336d4a0432308197a63e477e5aaeae6131d24cab70b06812854243f"} Apr 24 16:39:23.944123 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:23.943866 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" podStartSLOduration=2.943848508 podStartE2EDuration="2.943848508s" podCreationTimestamp="2026-04-24 16:39:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:39:22.933358968 +0000 UTC m=+3.682529729" watchObservedRunningTime="2026-04-24 16:39:23.943848508 +0000 UTC m=+4.693019260" Apr 24 16:39:24.888610 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.888580 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dx77w"] Apr 24 16:39:24.890937 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.890517 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:24.890937 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:24.890587 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:24.937328 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.936787 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"41c83e021f241dcae39ca57bc9bf5fb2","Type":"ContainerStarted","Data":"176c4582c1c0a788deaef078fdf8e7a6498fc2c2a6749ba7c7f26bae13f88a1e"} Apr 24 16:39:24.987603 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.987225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-kubelet-config\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:24.987603 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.987292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-dbus\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:24.987603 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:24.987335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.088805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.088376 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.088805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.088450 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-kubelet-config\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.088805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.088497 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-dbus\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.088805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.088761 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-kubelet-config\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.089110 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.088837 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:25.089110 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.088913 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:25.588892353 +0000 UTC m=+6.338063106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:25.089808 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.089277 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a768d2bb-7b4b-4040-94aa-1bf6370247c1-dbus\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.390985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.390917 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:25.391250 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.391229 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:25.391345 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.391327 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:29.391307683 +0000 UTC m=+10.140478436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:25.492542 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.492135 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:25.492542 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.492310 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:25.492542 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.492333 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:25.492542 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.492348 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:25.492542 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.492414 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:29.492393082 +0000 UTC m=+10.241563823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:25.592698 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.592650 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:25.592901 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.592831 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:25.592901 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.592899 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:26.592880169 +0000 UTC m=+7.342050919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:25.847994 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.847916 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:25.848140 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.848056 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:25.848437 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:25.848418 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:25.848528 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:25.848508 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:26.602367 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:26.602327 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:26.602815 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:26.602501 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:26.602815 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:26.602590 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:28.602571198 +0000 UTC m=+9.351741942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:26.845267 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:26.845238 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:26.845491 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:26.845330 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:27.846423 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:27.846394 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:27.846856 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:27.846560 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:27.846856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:27.846821 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:27.846947 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:27.846919 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:28.618657 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:28.618609 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:28.618822 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:28.618793 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:28.618900 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:28.618871 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:32.618851776 +0000 UTC m=+13.368022521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:28.845489 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:28.845450 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:28.845650 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:28.845591 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:29.425002 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:29.424954 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:29.425449 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.425125 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:29.425449 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.425202 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:37.425182876 +0000 UTC m=+18.174353629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:29.526404 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:29.526202 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:29.526404 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.526375 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:29.526404 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.526396 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:29.526404 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.526411 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:29.526737 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.526469 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:37.52645409 +0000 UTC m=+18.275624849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:29.846922 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:29.846847 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:29.847079 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.846977 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:29.847079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:29.846990 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:29.847181 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:29.847107 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:30.845851 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:30.845820 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:30.846228 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:30.845936 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:31.845997 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:31.845967 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:31.845997 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:31.845997 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:31.846420 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:31.846085 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:31.846420 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:31.846242 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:32.651221 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:32.651191 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:32.651406 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:32.651359 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:32.651457 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:32.651435 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:40.651413007 +0000 UTC m=+21.400583758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:32.845840 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:32.845809 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:32.845983 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:32.845929 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:33.846102 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:33.846061 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:33.846102 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:33.846085 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:33.846684 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:33.846207 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:33.846684 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:33.846290 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:34.846171 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:34.846140 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:34.846716 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:34.846254 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:35.846214 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:35.846180 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:35.846889 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:35.846399 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:35.846889 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:35.846515 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:35.846889 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:35.846572 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:36.846024 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:36.845990 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:36.846247 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:36.846110 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:37.486660 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:37.486612 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:37.486868 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.486791 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:37.486929 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.486874 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:53.486852129 +0000 UTC m=+34.236022879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:37.587809 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:37.587765 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:37.587985 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.587937 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:37.587985 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.587962 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:37.587985 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.587975 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:37.588106 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.588034 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:39:53.588020855 +0000 UTC m=+34.337191597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:37.845994 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:37.845882 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:37.845994 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:37.845899 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:37.846196 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.846008 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:37.846196 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:37.846106 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:38.845986 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:38.845686 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:38.845986 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:38.845808 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:38.961608 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:38.961566 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-srzmm" event={"ID":"813b1e82-26e4-4a64-a2d6-d4bc774b92e2","Type":"ContainerStarted","Data":"4a21fbe425b3b700a7547a2ca51ab77370f1244c8d468ee0350ad9684d5e864f"} Apr 24 16:39:39.846548 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.846173 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:39.846830 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.846233 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:39.847209 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:39.847147 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:39.847353 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:39.847039 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:39.964469 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.964392 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrlj6" event={"ID":"afd8262e-c6d9-4d7a-89c6-4b87b7281a67","Type":"ContainerStarted","Data":"9cb303bf74f200a0dfa213dca28668ccfd237fb5934299fd5beb86feae818241"} Apr 24 16:39:39.965753 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.965724 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q6scl" event={"ID":"da21c390-2975-4210-a740-ee0091b9c5b0","Type":"ContainerStarted","Data":"b34e9ae13f38f8a139e76e9bc535a826f70f86b677ae19679999a56da86c375b"} Apr 24 16:39:39.966888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.966868 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" event={"ID":"9bf0ebbf-8c09-4abd-b116-e82d4d4f8171","Type":"ContainerStarted","Data":"817fb5526cbda25a25d6e0c6c1147f71e402d776a681720ae91d0db3013625ba"} Apr 24 16:39:39.969193 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:39:39.969484 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969465 2561 generic.go:358] "Generic (PLEG): container finished" podID="dc972117-5b3e-4446-8204-6290e86329ad" containerID="b10d7a536af9215ec431086e111fe17db5c655ea697270f81162854c844a3bdf" exitCode=1 Apr 24 16:39:39.969484 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969477 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"34ebafe8dbe1e718c8d1c63924150a86491fada212d9beab3f9a9df579a76c8e"} Apr 24 16:39:39.969594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969497 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"07ad4092f547ef7d7dd84e86a010e92940da8ad0d7cfc395c2b379ac9144caee"} Apr 24 16:39:39.969594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969506 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"6ca7678446904d6f09abf1ee642314038cdbda730cc6d10b509acb53280f37de"} Apr 24 16:39:39.969594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969516 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"234ad8d88c92bf2a7bdd95a18858dd41c6ec29dcaf8607a73411fcd0234e04c2"} Apr 24 16:39:39.969594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969527 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerDied","Data":"b10d7a536af9215ec431086e111fe17db5c655ea697270f81162854c844a3bdf"} Apr 24 16:39:39.969594 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.969541 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"d40f71d7c8de0546748d6d4a8538a43a5ee8022f430e6d23b5b74a6eecfc943c"} Apr 24 16:39:39.970559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.970539 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4586" event={"ID":"2d1c3528-cf11-4ef1-bb88-2f59df7e45a9","Type":"ContainerStarted","Data":"d7a5f8786274ec2731a21332c5736e2875913e8b29379162930a94cf882a7bec"} Apr 24 16:39:39.971813 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.971793 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="d026df72a8711f483675beaea4b39d683ab7a8d2206323714e8acd3b6056cde6" exitCode=0 Apr 24 16:39:39.971901 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.971870 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"d026df72a8711f483675beaea4b39d683ab7a8d2206323714e8acd3b6056cde6"} Apr 24 16:39:39.972989 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.972974 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" event={"ID":"872ed659-a1a1-468f-8ab1-097f13c7e6eb","Type":"ContainerStarted","Data":"e871223a6d017201fdf66b4d65881936705dd8489cdaa2e12d9fb32b8cc2331b"} Apr 24 16:39:39.981040 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.981009 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" podStartSLOduration=18.98100004 podStartE2EDuration="18.98100004s" podCreationTimestamp="2026-04-24 16:39:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:39:24.976713026 +0000 UTC m=+5.725883785" watchObservedRunningTime="2026-04-24 16:39:39.98100004 +0000 UTC m=+20.730170801" Apr 24 16:39:39.981257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.981238 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-zrlj6" podStartSLOduration=3.509561159 podStartE2EDuration="19.981232714s" podCreationTimestamp="2026-04-24 16:39:20 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.32958722 +0000 UTC m=+3.078757960" lastFinishedPulling="2026-04-24 16:39:38.801258772 +0000 UTC m=+19.550429515" observedRunningTime="2026-04-24 16:39:39.980640667 +0000 UTC m=+20.729811428" watchObservedRunningTime="2026-04-24 16:39:39.981232714 +0000 UTC m=+20.730403475" Apr 24 16:39:39.997125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:39.994793 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-srzmm" podStartSLOduration=12.308009903 podStartE2EDuration="20.994778269s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.340366358 +0000 UTC m=+3.089537100" lastFinishedPulling="2026-04-24 16:39:31.027134713 +0000 UTC m=+11.776305466" observedRunningTime="2026-04-24 16:39:39.994260343 +0000 UTC m=+20.743431102" watchObservedRunningTime="2026-04-24 16:39:39.994778269 +0000 UTC m=+20.743949031" Apr 24 16:39:40.013585 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.013537 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7vwnx" podStartSLOduration=4.551748267 podStartE2EDuration="21.013524622s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.341022454 +0000 UTC m=+3.090193195" lastFinishedPulling="2026-04-24 16:39:38.802798811 +0000 UTC m=+19.551969550" observedRunningTime="2026-04-24 16:39:40.013323069 +0000 UTC m=+20.762493830" watchObservedRunningTime="2026-04-24 16:39:40.013524622 +0000 UTC m=+20.762695379" Apr 24 16:39:40.028052 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.027994 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4586" podStartSLOduration=4.561114285 podStartE2EDuration="21.027984s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.334483336 +0000 UTC m=+3.083654075" lastFinishedPulling="2026-04-24 16:39:38.801353037 +0000 UTC m=+19.550523790" observedRunningTime="2026-04-24 16:39:40.027789881 +0000 UTC m=+20.776960641" watchObservedRunningTime="2026-04-24 16:39:40.027984 +0000 UTC m=+20.777154804" Apr 24 16:39:40.064856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.064794 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-q6scl" podStartSLOduration=4.578597456 podStartE2EDuration="21.064780413s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.326795485 +0000 UTC m=+3.075966225" lastFinishedPulling="2026-04-24 16:39:38.812978428 +0000 UTC m=+19.562149182" observedRunningTime="2026-04-24 16:39:40.064729616 +0000 UTC m=+20.813900376" watchObservedRunningTime="2026-04-24 16:39:40.064780413 +0000 UTC m=+20.813951173" Apr 24 16:39:40.258638 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.258497 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 24 16:39:40.712574 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.712484 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:40.712765 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:40.712663 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:40.712765 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:40.712761 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret podName:a768d2bb-7b4b-4040-94aa-1bf6370247c1 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:56.712739838 +0000 UTC m=+37.461910595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret") pod "global-pull-secret-syncer-dx77w" (UID: "a768d2bb-7b4b-4040-94aa-1bf6370247c1") : object "kube-system"/"original-pull-secret" not registered Apr 24 16:39:40.828746 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.828555 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-24T16:39:40.258632709Z","UUID":"15223307-7c3f-4204-8027-edae5a9f3f37","Handler":null,"Name":"","Endpoint":""} Apr 24 16:39:40.831789 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.831735 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 24 16:39:40.831789 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.831764 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 24 16:39:40.846085 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.846059 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:40.846220 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:40.846194 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:40.976783 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.976684 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gx57q" event={"ID":"9c0d56c6-7fdc-4b45-ba58-6da3aaccac01","Type":"ContainerStarted","Data":"dde5a7d13342f688b93af8b38d53b86ae94295a49620c5d838c1fab9aff27fbe"} Apr 24 16:39:40.978832 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:40.978766 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" event={"ID":"872ed659-a1a1-468f-8ab1-097f13c7e6eb","Type":"ContainerStarted","Data":"1a2cadb0c4d0fcd5fcb91a93f5d8088a809d92c2e849d616b460c33c9aa4a640"} Apr 24 16:39:41.846201 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:41.846153 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:41.846390 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:41.846168 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:41.846390 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:41.846309 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:41.846497 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:41.846431 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:41.983255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:41.983157 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" event={"ID":"872ed659-a1a1-468f-8ab1-097f13c7e6eb","Type":"ContainerStarted","Data":"91cf709b72cfd4d7dfd432673837375176d520d161e7e3a40bee72be2a4fe93f"} Apr 24 16:39:41.986493 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:41.986432 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:39:41.986925 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:41.986900 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"faf25bad574b59b053ea3e9c35c704c277ba03ac2bf648ffabdc18906366c369"} Apr 24 16:39:42.000489 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:42.000449 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6rgr7" podStartSLOduration=4.104042944 podStartE2EDuration="23.000435692s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.338151152 +0000 UTC m=+3.087321896" lastFinishedPulling="2026-04-24 16:39:41.234543886 +0000 UTC m=+21.983714644" observedRunningTime="2026-04-24 16:39:42.0003515 +0000 UTC m=+22.749522279" watchObservedRunningTime="2026-04-24 16:39:42.000435692 +0000 UTC m=+22.749606467" Apr 24 16:39:42.001049 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:42.001011 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gx57q" podStartSLOduration=5.53264365 podStartE2EDuration="22.001000409s" podCreationTimestamp="2026-04-24 16:39:20 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.332891696 +0000 UTC m=+3.082062435" lastFinishedPulling="2026-04-24 16:39:38.801248442 +0000 UTC m=+19.550419194" observedRunningTime="2026-04-24 16:39:40.998468909 +0000 UTC m=+21.747639670" watchObservedRunningTime="2026-04-24 16:39:42.001000409 +0000 UTC m=+22.750171170" Apr 24 16:39:42.846106 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:42.846069 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:42.846291 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:42.846201 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:43.845470 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:43.845437 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:43.845470 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:43.845465 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:43.846326 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:43.845586 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:43.846326 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:43.845693 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:43.993657 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:43.993638 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:39:44.534307 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:44.534119 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:44.534772 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:44.534756 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:44.846306 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:44.846233 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:44.846725 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:44.846340 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:45.000790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.000765 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:39:45.001079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.001052 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"b0cfe124239bc62321d038b2999b22ac8a75ebe0c51c554b10d1f161d5727b9c"} Apr 24 16:39:45.001368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.001351 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:45.001518 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.001505 2561 scope.go:117] "RemoveContainer" containerID="b10d7a536af9215ec431086e111fe17db5c655ea697270f81162854c844a3bdf" Apr 24 16:39:45.002645 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.002628 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="d530d09f345005d8cc8203bf69916ff168a6f1e76868bdb892548a96de2fa9a6" exitCode=0 Apr 24 16:39:45.002743 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.002707 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"d530d09f345005d8cc8203bf69916ff168a6f1e76868bdb892548a96de2fa9a6"} Apr 24 16:39:45.017122 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.017105 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:45.845870 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.845834 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:45.846032 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:45.845834 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:45.846032 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:45.845939 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:45.846032 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:45.845998 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:46.007041 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.006913 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:39:46.007344 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.007307 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" event={"ID":"dc972117-5b3e-4446-8204-6290e86329ad","Type":"ContainerStarted","Data":"5be91c3057b8fbbd4e6efcdeffd0efd36454e594d64dffb04a118f5d673806bb"} Apr 24 16:39:46.007603 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.007583 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:46.007649 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.007617 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:46.021869 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.021846 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:39:46.045576 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.045523 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" podStartSLOduration=9.523598353 podStartE2EDuration="26.045506701s" podCreationTimestamp="2026-04-24 16:39:20 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.336305564 +0000 UTC m=+3.085476304" lastFinishedPulling="2026-04-24 16:39:38.858213897 +0000 UTC m=+19.607384652" observedRunningTime="2026-04-24 16:39:46.043349382 +0000 UTC m=+26.792520143" watchObservedRunningTime="2026-04-24 16:39:46.045506701 +0000 UTC m=+26.794677463" Apr 24 16:39:46.150082 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.150057 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhwg5"] Apr 24 16:39:46.150186 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.150175 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:46.150333 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:46.150312 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:46.155145 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.155115 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwzw9"] Apr 24 16:39:46.155267 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.155248 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:46.155473 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:46.155434 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:46.155921 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.155897 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dx77w"] Apr 24 16:39:46.156051 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:46.156005 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:46.156106 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:46.156088 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:47.010551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:47.010516 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="de6eb684097753d589a7d21734ff99404b21fc094c7b730e8a8dd0732053bdb6" exitCode=0 Apr 24 16:39:47.010943 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:47.010605 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"de6eb684097753d589a7d21734ff99404b21fc094c7b730e8a8dd0732053bdb6"} Apr 24 16:39:47.846300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:47.846262 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:47.846449 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:47.846372 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:47.846449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:47.846389 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:47.846449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:47.846403 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:47.846553 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:47.846470 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:47.846553 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:47.846527 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:48.029137 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:48.029082 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" podUID="dc972117-5b3e-4446-8204-6290e86329ad" containerName="ovnkube-controller" probeResult="failure" output="" Apr 24 16:39:48.897589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:48.897546 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:48.897762 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:48.897709 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Apr 24 16:39:48.898242 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:48.898225 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-srzmm" Apr 24 16:39:49.016997 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:49.016968 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="6f71ddc532e2aa91ce920fa9a004b1efadcdb611757d45f01e965b980dcf6fcd" exitCode=0 Apr 24 16:39:49.017140 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:49.017046 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"6f71ddc532e2aa91ce920fa9a004b1efadcdb611757d45f01e965b980dcf6fcd"} Apr 24 16:39:49.846345 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:49.846312 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:49.847156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:49.846392 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:49.847156 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:49.846528 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:49.847156 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:49.846529 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:49.847156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:49.846540 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:49.847156 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:49.846639 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:51.845715 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:51.845682 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:51.846278 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:51.845785 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dx77w" podUID="a768d2bb-7b4b-4040-94aa-1bf6370247c1" Apr 24 16:39:51.846278 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:51.845790 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:51.846278 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:51.845811 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:51.846278 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:51.845884 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:39:51.846278 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:51.845928 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhwg5" podUID="8f64b904-5057-4953-8d22-8b20d43e4fcf" Apr 24 16:39:52.030462 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.030387 2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeReady" Apr 24 16:39:52.030600 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.030530 2561 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 24 16:39:52.075758 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.075627 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jqv7x"] Apr 24 16:39:52.104219 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.104191 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h7m28"] Apr 24 16:39:52.104386 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.104364 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.106953 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.106927 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ljxll\"" Apr 24 16:39:52.106953 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.106929 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 24 16:39:52.107134 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.106933 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 24 16:39:52.120005 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.119978 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqv7x"] Apr 24 16:39:52.120005 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.120003 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h7m28"] Apr 24 16:39:52.120181 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.120106 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.122405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.122383 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 24 16:39:52.122405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.122388 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 24 16:39:52.122593 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.122569 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 24 16:39:52.122718 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.122697 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2lclw\"" Apr 24 16:39:52.197547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.197513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxqw\" (UniqueName: \"kubernetes.io/projected/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-kube-api-access-gsxqw\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.197547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.197544 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.197730 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.197582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-tmp-dir\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.197730 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.197621 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-config-volume\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.298958 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.298890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.298958 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.298936 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxqw\" (UniqueName: \"kubernetes.io/projected/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-kube-api-access-gsxqw\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.299164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.298968 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.299164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.298993 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5t5d\" (UniqueName: \"kubernetes.io/projected/780cdd19-26a2-4a6f-844d-748c9e024c5f-kube-api-access-x5t5d\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.299164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.299066 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-tmp-dir\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.299164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.299099 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-config-volume\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.299164 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.299106 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:39:52.299389 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.299220 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:52.799196473 +0000 UTC m=+33.548367234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:39:52.299474 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.299456 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-tmp-dir\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.299862 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.299841 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-config-volume\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.309502 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.309475 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxqw\" (UniqueName: \"kubernetes.io/projected/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-kube-api-access-gsxqw\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.399648 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.399609 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.399839 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.399681 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5t5d\" (UniqueName: \"kubernetes.io/projected/780cdd19-26a2-4a6f-844d-748c9e024c5f-kube-api-access-x5t5d\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.399839 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.399781 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:39:52.399924 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.399849 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:39:52.899830854 +0000 UTC m=+33.649001601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:39:52.415104 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.415079 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5t5d\" (UniqueName: \"kubernetes.io/projected/780cdd19-26a2-4a6f-844d-748c9e024c5f-kube-api-access-x5t5d\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.802916 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.802882 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:52.803090 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.803069 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:39:52.803153 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.803141 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:53.803124702 +0000 UTC m=+34.552295446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:39:52.903910 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:52.903877 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:52.904334 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.904059 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:39:52.904334 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:52.904132 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:39:53.904109989 +0000 UTC m=+34.653280733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:39:53.509511 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.509478 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:53.509779 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.509652 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:53.509779 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.509746 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:40:25.509726609 +0000 UTC m=+66.258897356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 24 16:39:53.610875 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.610839 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:53.611060 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.611009 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 24 16:39:53.611060 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.611028 2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 24 16:39:53.611060 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.611038 2561 projected.go:194] Error preparing data for projected volume kube-api-access-57l4w for pod openshift-network-diagnostics/network-check-target-mhwg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:53.611198 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.611117 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w podName:8f64b904-5057-4953-8d22-8b20d43e4fcf nodeName:}" failed. No retries permitted until 2026-04-24 16:40:25.611098859 +0000 UTC m=+66.360269602 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-57l4w" (UniqueName: "kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w") pod "network-check-target-mhwg5" (UID: "8f64b904-5057-4953-8d22-8b20d43e4fcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 24 16:39:53.812702 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.812597 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:53.812960 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.812797 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:39:53.812960 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.812879 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:55.812857714 +0000 UTC m=+36.562028457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:39:53.845894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.845860 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:39:53.846043 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.846005 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:39:53.846273 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.846204 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:53.849820 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849796 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ckwm\"" Apr 24 16:39:53.849932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849836 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 24 16:39:53.849932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849847 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 24 16:39:53.849932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849855 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czswq\"" Apr 24 16:39:53.849932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849802 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 24 16:39:53.849932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.849796 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 24 16:39:53.913162 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:53.913135 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:53.913544 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.913308 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:39:53.913544 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:53.913379 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:39:55.913359058 +0000 UTC m=+36.662529817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:39:55.031480 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:55.031449 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="56d8ab8587f2d008dd44bfe1a57c2877c6714e71bdc2e71acb2074c5caba500e" exitCode=0 Apr 24 16:39:55.031809 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:55.031505 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"56d8ab8587f2d008dd44bfe1a57c2877c6714e71bdc2e71acb2074c5caba500e"} Apr 24 16:39:55.829089 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:55.829000 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:55.829226 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:55.829153 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:39:55.829226 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:55.829219 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:39:59.829203443 +0000 UTC m=+40.578374183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:39:55.929864 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:55.929830 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:55.930017 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:55.929940 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:39:55.930017 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:55.930015 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:39:59.929998288 +0000 UTC m=+40.679169028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:39:56.036963 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.036933 2561 generic.go:358] "Generic (PLEG): container finished" podID="b21bb7b3-2bb8-4576-b27b-a786cd88c140" containerID="c0192a2f8db9949c855c9686c259a9e754dd496f7201848a235a0a26a1c5017d" exitCode=0 Apr 24 16:39:56.037325 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.036984 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerDied","Data":"c0192a2f8db9949c855c9686c259a9e754dd496f7201848a235a0a26a1c5017d"} Apr 24 16:39:56.735072 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.735028 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:56.738400 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.738379 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a768d2bb-7b4b-4040-94aa-1bf6370247c1-original-pull-secret\") pod \"global-pull-secret-syncer-dx77w\" (UID: \"a768d2bb-7b4b-4040-94aa-1bf6370247c1\") " pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:56.868761 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.868708 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dx77w" Apr 24 16:39:56.994410 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:56.994117 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dx77w"] Apr 24 16:39:57.039612 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:57.039583 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dx77w" event={"ID":"a768d2bb-7b4b-4040-94aa-1bf6370247c1","Type":"ContainerStarted","Data":"7d2c5c6b8872ba78e4e196a575a7e692d732ceb1669cb20a98d9957a69473261"} Apr 24 16:39:57.042355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:57.042331 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" event={"ID":"b21bb7b3-2bb8-4576-b27b-a786cd88c140","Type":"ContainerStarted","Data":"889fb1a942f254a71bf12b5b0a6a9bc98454e7951462787a11a3f1272d1a3f7d"} Apr 24 16:39:57.066788 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:57.066747 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qkvcg" podStartSLOduration=5.736589416 podStartE2EDuration="38.066735975s" podCreationTimestamp="2026-04-24 16:39:19 +0000 UTC" firstStartedPulling="2026-04-24 16:39:22.333921837 +0000 UTC m=+3.083092582" lastFinishedPulling="2026-04-24 16:39:54.664068163 +0000 UTC m=+35.413239141" observedRunningTime="2026-04-24 16:39:57.065768314 +0000 UTC m=+37.814939075" watchObservedRunningTime="2026-04-24 16:39:57.066735975 +0000 UTC m=+37.815906735" Apr 24 16:39:59.860015 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:59.859979 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:39:59.860492 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:59.860104 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:39:59.860492 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:59.860157 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:40:07.860144048 +0000 UTC m=+48.609314787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:39:59.960677 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:39:59.960639 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:39:59.960861 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:59.960839 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:39:59.960937 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:39:59.960922 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:40:07.960897963 +0000 UTC m=+48.710068716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:40:01.052002 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:01.051923 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dx77w" event={"ID":"a768d2bb-7b4b-4040-94aa-1bf6370247c1","Type":"ContainerStarted","Data":"b545e84d2190dd387ee475b6d222abaa350b5bf675326a2f443dd67485961213"} Apr 24 16:40:01.067224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:01.067181 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dx77w" podStartSLOduration=33.411139369 podStartE2EDuration="37.067167644s" podCreationTimestamp="2026-04-24 16:39:24 +0000 UTC" firstStartedPulling="2026-04-24 16:39:57.001922223 +0000 UTC m=+37.751092962" lastFinishedPulling="2026-04-24 16:40:00.657950495 +0000 UTC m=+41.407121237" observedRunningTime="2026-04-24 16:40:01.066575609 +0000 UTC m=+41.815746371" watchObservedRunningTime="2026-04-24 16:40:01.067167644 +0000 UTC m=+41.816338405" Apr 24 16:40:07.912871 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:07.912838 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:40:07.913228 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:07.912949 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:40:07.913228 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:07.912999 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:40:23.91298657 +0000 UTC m=+64.662157309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:40:08.013279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:08.013243 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:40:08.013395 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:08.013375 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:40:08.013455 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:08.013446 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:40:24.013431519 +0000 UTC m=+64.762602258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:40:18.022707 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:18.022657 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxfbv" Apr 24 16:40:23.919353 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:23.919301 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:40:23.919857 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:23.919461 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:40:23.919857 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:23.919542 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:40:55.919521399 +0000 UTC m=+96.668692139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:40:24.020577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:24.020550 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:40:24.020706 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:24.020645 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:40:24.020747 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:24.020710 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:40:56.020696279 +0000 UTC m=+96.769867019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:40:25.530893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.530844 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:40:25.533718 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.533700 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 24 16:40:25.541778 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:25.541762 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 24 16:40:25.541860 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:25.541820 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:29.54180416 +0000 UTC m=+130.290974899 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : secret "metrics-daemon-secret" not found Apr 24 16:40:25.631542 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.631498 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:40:25.634090 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.634068 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 24 16:40:25.643881 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.643858 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 24 16:40:25.655582 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.655562 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57l4w\" (UniqueName: \"kubernetes.io/projected/8f64b904-5057-4953-8d22-8b20d43e4fcf-kube-api-access-57l4w\") pod \"network-check-target-mhwg5\" (UID: \"8f64b904-5057-4953-8d22-8b20d43e4fcf\") " pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:40:25.659283 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.659265 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7ckwm\"" Apr 24 16:40:25.668154 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.668139 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:40:25.790944 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:25.790866 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhwg5"] Apr 24 16:40:25.795964 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:40:25.794462 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f64b904_5057_4953_8d22_8b20d43e4fcf.slice/crio-e4fc4569399bd4e91786afbb71cbb47d2f1550450f3499a422d31b50880c37e9 WatchSource:0}: Error finding container e4fc4569399bd4e91786afbb71cbb47d2f1550450f3499a422d31b50880c37e9: Status 404 returned error can't find the container with id e4fc4569399bd4e91786afbb71cbb47d2f1550450f3499a422d31b50880c37e9 Apr 24 16:40:26.097904 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:26.097822 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhwg5" event={"ID":"8f64b904-5057-4953-8d22-8b20d43e4fcf","Type":"ContainerStarted","Data":"e4fc4569399bd4e91786afbb71cbb47d2f1550450f3499a422d31b50880c37e9"} Apr 24 16:40:29.105146 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:29.105113 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhwg5" event={"ID":"8f64b904-5057-4953-8d22-8b20d43e4fcf","Type":"ContainerStarted","Data":"0460c5d6dcc8f9682034f96c9c949c87ced72eaeea53739eb973becfe5aa8099"} Apr 24 16:40:29.105542 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:29.105259 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:40:29.120808 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:29.120765 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mhwg5" podStartSLOduration=66.639642321 podStartE2EDuration="1m9.120753571s" podCreationTimestamp="2026-04-24 16:39:20 +0000 UTC" firstStartedPulling="2026-04-24 16:40:25.797833822 +0000 UTC m=+66.547004568" lastFinishedPulling="2026-04-24 16:40:28.278945079 +0000 UTC m=+69.028115818" observedRunningTime="2026-04-24 16:40:29.120088095 +0000 UTC m=+69.869258855" watchObservedRunningTime="2026-04-24 16:40:29.120753571 +0000 UTC m=+69.869924324" Apr 24 16:40:55.942011 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:55.941956 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:40:55.942401 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:55.942100 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 24 16:40:55.942401 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:55.942168 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls podName:9d591d2a-71f2-4e1f-9e37-a4a1756a08f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:59.942152508 +0000 UTC m=+160.691323253 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls") pod "dns-default-jqv7x" (UID: "9d591d2a-71f2-4e1f-9e37-a4a1756a08f3") : secret "dns-default-metrics-tls" not found Apr 24 16:40:56.043084 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:40:56.043057 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:40:56.043230 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:56.043168 2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 24 16:40:56.043230 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:40:56.043220 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert podName:780cdd19-26a2-4a6f-844d-748c9e024c5f nodeName:}" failed. No retries permitted until 2026-04-24 16:42:00.043207697 +0000 UTC m=+160.792378437 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert") pod "ingress-canary-h7m28" (UID: "780cdd19-26a2-4a6f-844d-748c9e024c5f") : secret "canary-serving-cert" not found Apr 24 16:41:00.108812 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:00.108780 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mhwg5" Apr 24 16:41:29.562701 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:29.562646 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:41:29.563062 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:29.562789 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 24 16:41:29.563062 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:29.562878 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs podName:b71bdf81-e127-49a6-aca6-ff4846028fd9 nodeName:}" failed. No retries permitted until 2026-04-24 16:43:31.56286182 +0000 UTC m=+252.312032564 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs") pod "network-metrics-daemon-mwzw9" (UID: "b71bdf81-e127-49a6-aca6-ff4846028fd9") : secret "metrics-daemon-secret" not found Apr 24 16:41:30.452041 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.452006 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4"] Apr 24 16:41:30.456600 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.456576 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7c6f5d86d4-87vd7"] Apr 24 16:41:30.456768 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.456749 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.459435 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.459406 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Apr 24 16:41:30.459698 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.459495 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.459884 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.459858 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n4php\"" Apr 24 16:41:30.459884 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.459872 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Apr 24 16:41:30.460101 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.460085 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Apr 24 16:41:30.462365 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462321 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 24 16:41:30.462577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462560 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 24 16:41:30.462704 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462585 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 24 16:41:30.462704 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462585 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 24 16:41:30.462704 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462621 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fbnp\"" Apr 24 16:41:30.462704 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462644 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 24 16:41:30.462916 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.462839 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 24 16:41:30.468322 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.468298 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4"] Apr 24 16:41:30.484836 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.484793 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c6f5d86d4-87vd7"] Apr 24 16:41:30.553045 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.553012 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4"] Apr 24 16:41:30.556024 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.556009 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.559111 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.559091 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-czldl\"" Apr 24 16:41:30.559780 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.559762 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Apr 24 16:41:30.559953 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.559941 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Apr 24 16:41:30.569976 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.569955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.569984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqxg\" (UniqueName: \"kubernetes.io/projected/15b66498-b5b4-403e-8f0c-6754367fa215-kube-api-access-8qqxg\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.570023 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.570115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-default-certificate\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.570241 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.570287 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrcf4\" (UniqueName: \"kubernetes.io/projected/04a56f5c-eab6-4b94-9c0f-c771218f58f7-kube-api-access-vrcf4\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.570327 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.570316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-stats-auth\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.571710 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.571686 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4"] Apr 24 16:41:30.658982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.658951 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:41:30.661865 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.661848 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.664757 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.664735 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 24 16:41:30.664888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.664773 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t5hh5\"" Apr 24 16:41:30.664888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.664779 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 24 16:41:30.665106 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.665090 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 24 16:41:30.670331 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670310 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 24 16:41:30.670610 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.670703 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670631 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/142e9821-410f-44b3-9366-6e7f6419ac79-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.670773 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670728 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.670773 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670762 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vrcf4\" (UniqueName: \"kubernetes.io/projected/04a56f5c-eab6-4b94-9c0f-c771218f58f7-kube-api-access-vrcf4\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.670867 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670790 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-stats-auth\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.670867 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.670813 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:30.670867 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670826 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.670867 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.670855 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqxg\" (UniqueName: \"kubernetes.io/projected/15b66498-b5b4-403e-8f0c-6754367fa215-kube-api-access-8qqxg\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.671049 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.670874 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:31.170854758 +0000 UTC m=+131.920025514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:30.671049 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.671021 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.671049 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.671043 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:31.171025431 +0000 UTC m=+131.920196176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:30.671203 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.671084 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 24 16:41:30.671203 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.671128 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls podName:04a56f5c-eab6-4b94-9c0f-c771218f58f7 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:31.171110371 +0000 UTC m=+131.920281116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-29cj4" (UID: "04a56f5c-eab6-4b94-9c0f-c771218f58f7") : secret "samples-operator-tls" not found Apr 24 16:41:30.671203 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.671124 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-default-certificate\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.673383 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.673359 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:41:30.673839 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.673823 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-stats-auth\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.673912 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.673855 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-default-certificate\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.678213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.678189 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrcf4\" (UniqueName: \"kubernetes.io/projected/04a56f5c-eab6-4b94-9c0f-c771218f58f7-kube-api-access-vrcf4\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:30.682375 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.682354 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqxg\" (UniqueName: \"kubernetes.io/projected/15b66498-b5b4-403e-8f0c-6754367fa215-kube-api-access-8qqxg\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:30.771798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771708 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.771798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.771798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771782 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/142e9821-410f-44b3-9366-6e7f6419ac79-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.772009 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772009 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.771842 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:30.772009 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.771921 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:31.271903644 +0000 UTC m=+132.021074384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:30.772009 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772009 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.771982 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.772016 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.772073 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5g7s\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.772102 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.772122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.772478 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.772459 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/142e9821-410f-44b3-9366-6e7f6419ac79-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:30.872525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872486 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872533 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872585 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872606 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872633 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872722 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l5g7s\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.872741 2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872749 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.872758 2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d8b869f-nzxwd: secret "image-registry-tls" not found Apr 24 16:41:30.872798 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.872778 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.873160 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:30.872815 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls podName:72164181-861f-46e6-9c99-34e40609fa17 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:31.372795025 +0000 UTC m=+132.121965782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls") pod "image-registry-558d8b869f-nzxwd" (UID: "72164181-861f-46e6-9c99-34e40609fa17") : secret "image-registry-tls" not found Apr 24 16:41:30.873656 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.873626 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.873932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.873910 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.874401 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.874378 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.875268 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.875252 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.875352 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.875284 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.883504 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.883484 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:30.884359 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:30.884342 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5g7s\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:31.175503 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:31.175455 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:31.175663 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:31.175523 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:31.175663 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.175621 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 24 16:41:31.175663 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.175642 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:32.175623981 +0000 UTC m=+132.924794740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:31.175663 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:31.175661 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:31.175826 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.175696 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls podName:04a56f5c-eab6-4b94-9c0f-c771218f58f7 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:32.175660497 +0000 UTC m=+132.924831236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-29cj4" (UID: "04a56f5c-eab6-4b94-9c0f-c771218f58f7") : secret "samples-operator-tls" not found Apr 24 16:41:31.175826 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.175756 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:31.175826 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.175794 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:32.175783164 +0000 UTC m=+132.924953904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:31.276221 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:31.276187 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:31.276451 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.276329 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:31.276451 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.276385 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:32.27637167 +0000 UTC m=+133.025542409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:31.376569 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:31.376536 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:31.376770 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.376685 2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 24 16:41:31.376770 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.376701 2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d8b869f-nzxwd: secret "image-registry-tls" not found Apr 24 16:41:31.376770 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:31.376766 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls podName:72164181-861f-46e6-9c99-34e40609fa17 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:32.376751779 +0000 UTC m=+133.125922523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls") pod "image-registry-558d8b869f-nzxwd" (UID: "72164181-861f-46e6-9c99-34e40609fa17") : secret "image-registry-tls" not found Apr 24 16:41:32.181462 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:32.181431 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:32.181474 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:32.181537 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.181576 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:34.181559381 +0000 UTC m=+134.930730120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.181623 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.181656 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.181700 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:34.18168417 +0000 UTC m=+134.930854926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:32.181891 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.181727 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls podName:04a56f5c-eab6-4b94-9c0f-c771218f58f7 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:34.181710035 +0000 UTC m=+134.930880775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-29cj4" (UID: "04a56f5c-eab6-4b94-9c0f-c771218f58f7") : secret "samples-operator-tls" not found Apr 24 16:41:32.282410 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:32.282376 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:32.282565 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.282507 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:32.282565 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.282561 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:34.282547742 +0000 UTC m=+135.031718486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:32.383384 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:32.383352 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:32.383512 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.383494 2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 24 16:41:32.383552 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.383514 2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d8b869f-nzxwd: secret "image-registry-tls" not found Apr 24 16:41:32.383586 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:32.383568 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls podName:72164181-861f-46e6-9c99-34e40609fa17 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:34.383554866 +0000 UTC m=+135.132725605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls") pod "image-registry-558d8b869f-nzxwd" (UID: "72164181-861f-46e6-9c99-34e40609fa17") : secret "image-registry-tls" not found Apr 24 16:41:34.196300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:34.196267 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:34.196300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:34.196307 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:34.196344 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.196418 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.196437 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:38.196420516 +0000 UTC m=+138.945591256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.196470 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.196472 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:38.196456706 +0000 UTC m=+138.945627462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:34.196707 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.196516 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls podName:04a56f5c-eab6-4b94-9c0f-c771218f58f7 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:38.196508374 +0000 UTC m=+138.945679114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-29cj4" (UID: "04a56f5c-eab6-4b94-9c0f-c771218f58f7") : secret "samples-operator-tls" not found Apr 24 16:41:34.297125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:34.297081 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:34.297285 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.297227 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:34.297321 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.297291 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:38.297276204 +0000 UTC m=+139.046446943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:34.398199 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:34.398164 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:34.398357 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.398335 2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 24 16:41:34.398447 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.398361 2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d8b869f-nzxwd: secret "image-registry-tls" not found Apr 24 16:41:34.398447 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:34.398423 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls podName:72164181-861f-46e6-9c99-34e40609fa17 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:38.398405651 +0000 UTC m=+139.147576413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls") pod "image-registry-558d8b869f-nzxwd" (UID: "72164181-861f-46e6-9c99-34e40609fa17") : secret "image-registry-tls" not found Apr 24 16:41:38.229115 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.229070 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:38.229115 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.229122 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.229240 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.229291 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:46.22926601 +0000 UTC m=+146.978436763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.229316 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls podName:04a56f5c-eab6-4b94-9c0f-c771218f58f7 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:46.229308769 +0000 UTC m=+146.978479508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls") pod "cluster-samples-operator-6dc5bdb6b4-29cj4" (UID: "04a56f5c-eab6-4b94-9c0f-c771218f58f7") : secret "samples-operator-tls" not found Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.229336 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.229417 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:38.229593 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.229458 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:46.229447 +0000 UTC m=+146.978617741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:38.330179 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.330151 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:38.330312 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.330278 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:38.330351 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.330342 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:46.33032724 +0000 UTC m=+147.079497979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:38.431141 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.431114 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:38.431250 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.431234 2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 24 16:41:38.431250 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.431249 2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d8b869f-nzxwd: secret "image-registry-tls" not found Apr 24 16:41:38.431335 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:38.431302 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls podName:72164181-861f-46e6-9c99-34e40609fa17 nodeName:}" failed. No retries permitted until 2026-04-24 16:41:46.431288909 +0000 UTC m=+147.180459648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls") pod "image-registry-558d8b869f-nzxwd" (UID: "72164181-861f-46e6-9c99-34e40609fa17") : secret "image-registry-tls" not found Apr 24 16:41:38.643827 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:38.643802 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrlj6_afd8262e-c6d9-4d7a-89c6-4b87b7281a67/dns-node-resolver/0.log" Apr 24 16:41:39.243405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:39.243376 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4586_2d1c3528-cf11-4ef1-bb88-2f59df7e45a9/node-ca/0.log" Apr 24 16:41:46.286579 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.286546 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:46.286579 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.286591 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:46.287005 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.286706 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:41:46.287005 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:46.286733 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:02.28671318 +0000 UTC m=+163.035883941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : configmap references non-existent config key: service-ca.crt Apr 24 16:41:46.287005 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:46.286786 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 24 16:41:46.287005 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:46.286852 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs podName:15b66498-b5b4-403e-8f0c-6754367fa215 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:02.286836699 +0000 UTC m=+163.036007444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs") pod "router-default-7c6f5d86d4-87vd7" (UID: "15b66498-b5b4-403e-8f0c-6754367fa215") : secret "router-metrics-certs-default" not found Apr 24 16:41:46.288977 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.288956 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/04a56f5c-eab6-4b94-9c0f-c771218f58f7-samples-operator-tls\") pod \"cluster-samples-operator-6dc5bdb6b4-29cj4\" (UID: \"04a56f5c-eab6-4b94-9c0f-c771218f58f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:46.366913 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.366882 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" Apr 24 16:41:46.387896 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.387865 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:41:46.388057 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:46.388035 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 24 16:41:46.388147 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:46.388135 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert podName:142e9821-410f-44b3-9366-6e7f6419ac79 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:02.388110767 +0000 UTC m=+163.137281510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-6c8j4" (UID: "142e9821-410f-44b3-9366-6e7f6419ac79") : secret "networking-console-plugin-cert" not found Apr 24 16:41:46.478843 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.478812 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4"] Apr 24 16:41:46.488920 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.488894 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:46.490980 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.490962 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"image-registry-558d8b869f-nzxwd\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:46.571409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.571346 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:46.685287 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:46.685253 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:41:46.688193 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:41:46.688162 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72164181_861f_46e6_9c99_34e40609fa17.slice/crio-45d0a44cde771a40ec683cf1543bf598b566497673330ff6afca4ecf5c2a7936 WatchSource:0}: Error finding container 45d0a44cde771a40ec683cf1543bf598b566497673330ff6afca4ecf5c2a7936: Status 404 returned error can't find the container with id 45d0a44cde771a40ec683cf1543bf598b566497673330ff6afca4ecf5c2a7936 Apr 24 16:41:47.255611 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:47.255578 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" event={"ID":"04a56f5c-eab6-4b94-9c0f-c771218f58f7","Type":"ContainerStarted","Data":"8f222e6ed090ba02800be7004678839f9a67bf5a433152db5e6975c9dfc89a02"} Apr 24 16:41:47.257069 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:47.257035 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" event={"ID":"72164181-861f-46e6-9c99-34e40609fa17","Type":"ContainerStarted","Data":"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689"} Apr 24 16:41:47.257200 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:47.257073 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" event={"ID":"72164181-861f-46e6-9c99-34e40609fa17","Type":"ContainerStarted","Data":"45d0a44cde771a40ec683cf1543bf598b566497673330ff6afca4ecf5c2a7936"} Apr 24 16:41:47.257200 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:47.257170 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:41:47.277395 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:47.277343 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" podStartSLOduration=17.277325255 podStartE2EDuration="17.277325255s" podCreationTimestamp="2026-04-24 16:41:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:41:47.276338947 +0000 UTC m=+148.025509710" watchObservedRunningTime="2026-04-24 16:41:47.277325255 +0000 UTC m=+148.026496014" Apr 24 16:41:48.263868 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:48.263833 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" event={"ID":"04a56f5c-eab6-4b94-9c0f-c771218f58f7","Type":"ContainerStarted","Data":"35731cbd402b1ed2b5f6e3b718fd1890526a560aae2c8ad733456c5c326000de"} Apr 24 16:41:48.264279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:48.263877 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" event={"ID":"04a56f5c-eab6-4b94-9c0f-c771218f58f7","Type":"ContainerStarted","Data":"506590791cfe3dafab1e879943d27ede56d3b21af35cd2d6825bbea7822ce65b"} Apr 24 16:41:48.283017 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:48.282967 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6dc5bdb6b4-29cj4" podStartSLOduration=16.67634564 podStartE2EDuration="18.282924141s" podCreationTimestamp="2026-04-24 16:41:30 +0000 UTC" firstStartedPulling="2026-04-24 16:41:46.517438556 +0000 UTC m=+147.266609295" lastFinishedPulling="2026-04-24 16:41:48.124017056 +0000 UTC m=+148.873187796" observedRunningTime="2026-04-24 16:41:48.282006916 +0000 UTC m=+149.031177676" watchObservedRunningTime="2026-04-24 16:41:48.282924141 +0000 UTC m=+149.032094902" Apr 24 16:41:55.115489 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:55.115451 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jqv7x" podUID="9d591d2a-71f2-4e1f-9e37-a4a1756a08f3" Apr 24 16:41:55.129589 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:55.129564 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-h7m28" podUID="780cdd19-26a2-4a6f-844d-748c9e024c5f" Apr 24 16:41:55.281318 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:55.281291 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:41:55.281461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:55.281302 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqv7x" Apr 24 16:41:56.862964 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:41:56.862926 2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mwzw9" podUID="b71bdf81-e127-49a6-aca6-ff4846028fd9" Apr 24 16:41:59.997058 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:59.997017 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:41:59.999335 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:41:59.999311 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d591d2a-71f2-4e1f-9e37-a4a1756a08f3-metrics-tls\") pod \"dns-default-jqv7x\" (UID: \"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3\") " pod="openshift-dns/dns-default-jqv7x" Apr 24 16:42:00.084620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.084590 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ljxll\"" Apr 24 16:42:00.092568 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.092552 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jqv7x" Apr 24 16:42:00.097347 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.097325 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:42:00.099621 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.099600 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/780cdd19-26a2-4a6f-844d-748c9e024c5f-cert\") pod \"ingress-canary-h7m28\" (UID: \"780cdd19-26a2-4a6f-844d-748c9e024c5f\") " pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:42:00.205331 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.205302 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jqv7x"] Apr 24 16:42:00.209134 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:00.209105 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d591d2a_71f2_4e1f_9e37_a4a1756a08f3.slice/crio-8214f7474914586f4468f9b16e12e2d1c1722fd137b99ad19b52462223aac140 WatchSource:0}: Error finding container 8214f7474914586f4468f9b16e12e2d1c1722fd137b99ad19b52462223aac140: Status 404 returned error can't find the container with id 8214f7474914586f4468f9b16e12e2d1c1722fd137b99ad19b52462223aac140 Apr 24 16:42:00.293178 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.293094 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqv7x" event={"ID":"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3","Type":"ContainerStarted","Data":"8214f7474914586f4468f9b16e12e2d1c1722fd137b99ad19b52462223aac140"} Apr 24 16:42:00.385468 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.385437 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2lclw\"" Apr 24 16:42:00.393458 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.393440 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h7m28" Apr 24 16:42:00.501994 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:00.501965 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h7m28"] Apr 24 16:42:00.505925 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:00.505902 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780cdd19_26a2_4a6f_844d_748c9e024c5f.slice/crio-e2952bf0883c4f76c7a1256b89c90198405d6d388c03d43319ef445cacb0380e WatchSource:0}: Error finding container e2952bf0883c4f76c7a1256b89c90198405d6d388c03d43319ef445cacb0380e: Status 404 returned error can't find the container with id e2952bf0883c4f76c7a1256b89c90198405d6d388c03d43319ef445cacb0380e Apr 24 16:42:01.297683 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.297625 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h7m28" event={"ID":"780cdd19-26a2-4a6f-844d-748c9e024c5f","Type":"ContainerStarted","Data":"e2952bf0883c4f76c7a1256b89c90198405d6d388c03d43319ef445cacb0380e"} Apr 24 16:42:01.431603 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.431572 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xlfzt"] Apr 24 16:42:01.434858 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.434837 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.437414 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.437388 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 24 16:42:01.437522 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.437509 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 24 16:42:01.438572 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.438554 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 24 16:42:01.438650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.438599 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jgncl\"" Apr 24 16:42:01.438650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.438626 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 24 16:42:01.444349 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.444327 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xlfzt"] Apr 24 16:42:01.484553 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.484529 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:42:01.489525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.489443 2561 patch_prober.go:28] interesting pod/image-registry-558d8b869f-nzxwd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Apr 24 16:42:01.489525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.489500 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 24 16:42:01.525153 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.525127 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8b56f5b5b-jclzv"] Apr 24 16:42:01.528531 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.528511 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.543469 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.543419 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8b56f5b5b-jclzv"] Apr 24 16:42:01.608914 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.608878 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d427b7d-8313-4e70-8dd1-f8ae2049538f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.609035 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.608968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d427b7d-8313-4e70-8dd1-f8ae2049538f-crio-socket\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.609035 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.608996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9q9\" (UniqueName: \"kubernetes.io/projected/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-api-access-mh9q9\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.609035 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.609024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d427b7d-8313-4e70-8dd1-f8ae2049538f-data-volume\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.609194 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.609071 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.710550 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-certificates\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710754 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-tls\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710754 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710586 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-trusted-ca\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710754 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-ca-trust-extracted\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710754 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-image-registry-private-configuration\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710778 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d427b7d-8313-4e70-8dd1-f8ae2049538f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710842 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d427b7d-8313-4e70-8dd1-f8ae2049538f-crio-socket\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710860 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9q9\" (UniqueName: \"kubernetes.io/projected/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-api-access-mh9q9\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-bound-sa-token\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710912 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d427b7d-8313-4e70-8dd1-f8ae2049538f-data-volume\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-installation-pull-secrets\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.710956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710937 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d427b7d-8313-4e70-8dd1-f8ae2049538f-crio-socket\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.711232 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.710960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgs4\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-kube-api-access-7bgs4\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.711232 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.711024 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.711484 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.711460 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d427b7d-8313-4e70-8dd1-f8ae2049538f-data-volume\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.711646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.711619 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.713429 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.713404 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d427b7d-8313-4e70-8dd1-f8ae2049538f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.718767 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.718718 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9q9\" (UniqueName: \"kubernetes.io/projected/9d427b7d-8313-4e70-8dd1-f8ae2049538f-kube-api-access-mh9q9\") pod \"insights-runtime-extractor-xlfzt\" (UID: \"9d427b7d-8313-4e70-8dd1-f8ae2049538f\") " pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.747064 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.747041 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xlfzt" Apr 24 16:42:01.811833 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.811801 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-ca-trust-extracted\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.811998 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.811840 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-image-registry-private-configuration\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.811998 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.811901 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-bound-sa-token\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.811998 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.811932 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-installation-pull-secrets\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.811998 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.811962 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgs4\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-kube-api-access-7bgs4\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.812195 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.812010 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-certificates\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.812195 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.812035 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-tls\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.812330 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.812288 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-ca-trust-extracted\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.812449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.812426 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-trusted-ca\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.812978 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.812926 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-certificates\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.813403 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.813379 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-trusted-ca\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.814830 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.814784 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-installation-pull-secrets\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.814907 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.814873 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-registry-tls\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.815081 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.815037 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-image-registry-private-configuration\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.820834 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.820765 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgs4\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-kube-api-access-7bgs4\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.821091 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.821064 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c35319be-d3a9-48a8-9aa6-3e1d55bccb11-bound-sa-token\") pod \"image-registry-8b56f5b5b-jclzv\" (UID: \"c35319be-d3a9-48a8-9aa6-3e1d55bccb11\") " pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:01.840873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:01.840842 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:02.149069 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.149043 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xlfzt"] Apr 24 16:42:02.152431 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:02.152407 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d427b7d_8313_4e70_8dd1_f8ae2049538f.slice/crio-ac1a91e57250ff12a5192313e27f26f2d9fef2b2d518a5a386adb67c8a574e8a WatchSource:0}: Error finding container ac1a91e57250ff12a5192313e27f26f2d9fef2b2d518a5a386adb67c8a574e8a: Status 404 returned error can't find the container with id ac1a91e57250ff12a5192313e27f26f2d9fef2b2d518a5a386adb67c8a574e8a Apr 24 16:42:02.170596 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.170568 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8b56f5b5b-jclzv"] Apr 24 16:42:02.174932 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:02.174910 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35319be_d3a9_48a8_9aa6_3e1d55bccb11.slice/crio-49df1592b1b64e84e3d42e5c05ffc5c22e5df6273b223b23e09ce7a9cdef79f9 WatchSource:0}: Error finding container 49df1592b1b64e84e3d42e5c05ffc5c22e5df6273b223b23e09ce7a9cdef79f9: Status 404 returned error can't find the container with id 49df1592b1b64e84e3d42e5c05ffc5c22e5df6273b223b23e09ce7a9cdef79f9 Apr 24 16:42:02.302636 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.302583 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h7m28" event={"ID":"780cdd19-26a2-4a6f-844d-748c9e024c5f","Type":"ContainerStarted","Data":"5a3111bc99b5255a0cf6125383388f6831775aced71aac7a23b41cca7b47471e"} Apr 24 16:42:02.303924 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.303886 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" event={"ID":"c35319be-d3a9-48a8-9aa6-3e1d55bccb11","Type":"ContainerStarted","Data":"41d7faf170f86688ae440e456c952feb3883c1bbe029c2758633a489f01e2a28"} Apr 24 16:42:02.303924 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.303914 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" event={"ID":"c35319be-d3a9-48a8-9aa6-3e1d55bccb11","Type":"ContainerStarted","Data":"49df1592b1b64e84e3d42e5c05ffc5c22e5df6273b223b23e09ce7a9cdef79f9"} Apr 24 16:42:02.304135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.304008 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:02.305649 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.305612 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqv7x" event={"ID":"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3","Type":"ContainerStarted","Data":"fced4dc86e0adc5a78ef18a93100e96ffa3b7655e1148f38f17264fa9275069d"} Apr 24 16:42:02.305773 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.305647 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jqv7x" event={"ID":"9d591d2a-71f2-4e1f-9e37-a4a1756a08f3","Type":"ContainerStarted","Data":"b8275c58a4fd0422f6d6f16b8b5f9d811a53d206ad3ca5ddb18c7f080b6ff7b7"} Apr 24 16:42:02.305773 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.305684 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jqv7x" Apr 24 16:42:02.307003 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.306981 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xlfzt" event={"ID":"9d427b7d-8313-4e70-8dd1-f8ae2049538f","Type":"ContainerStarted","Data":"1ad7d86405ef1b9d80610917bb2e2c46d19700be5db5ec8867d1fbf0c8a2d21f"} Apr 24 16:42:02.307116 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.307007 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xlfzt" event={"ID":"9d427b7d-8313-4e70-8dd1-f8ae2049538f","Type":"ContainerStarted","Data":"ac1a91e57250ff12a5192313e27f26f2d9fef2b2d518a5a386adb67c8a574e8a"} Apr 24 16:42:02.317537 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.317501 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:02.317658 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.317554 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:02.318386 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.318336 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15b66498-b5b4-403e-8f0c-6754367fa215-service-ca-bundle\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:02.319946 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.319900 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h7m28" podStartSLOduration=128.755190433 podStartE2EDuration="2m10.319886643s" podCreationTimestamp="2026-04-24 16:39:52 +0000 UTC" firstStartedPulling="2026-04-24 16:42:00.507756762 +0000 UTC m=+161.256927501" lastFinishedPulling="2026-04-24 16:42:02.072452958 +0000 UTC m=+162.821623711" observedRunningTime="2026-04-24 16:42:02.319059694 +0000 UTC m=+163.068230472" watchObservedRunningTime="2026-04-24 16:42:02.319886643 +0000 UTC m=+163.069057400" Apr 24 16:42:02.320268 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.320248 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b66498-b5b4-403e-8f0c-6754367fa215-metrics-certs\") pod \"router-default-7c6f5d86d4-87vd7\" (UID: \"15b66498-b5b4-403e-8f0c-6754367fa215\") " pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:02.337419 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.337328 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-jqv7x" podStartSLOduration=129.062950049 podStartE2EDuration="2m10.337311409s" podCreationTimestamp="2026-04-24 16:39:52 +0000 UTC" firstStartedPulling="2026-04-24 16:42:00.210766455 +0000 UTC m=+160.959937194" lastFinishedPulling="2026-04-24 16:42:01.485127813 +0000 UTC m=+162.234298554" observedRunningTime="2026-04-24 16:42:02.336535801 +0000 UTC m=+163.085706563" watchObservedRunningTime="2026-04-24 16:42:02.337311409 +0000 UTC m=+163.086482170" Apr 24 16:42:02.355353 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.355307 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" podStartSLOduration=1.355294182 podStartE2EDuration="1.355294182s" podCreationTimestamp="2026-04-24 16:42:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:42:02.354500842 +0000 UTC m=+163.103671603" watchObservedRunningTime="2026-04-24 16:42:02.355294182 +0000 UTC m=+163.104464945" Apr 24 16:42:02.419072 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.419043 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:42:02.421421 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.421395 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/142e9821-410f-44b3-9366-6e7f6419ac79-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-6c8j4\" (UID: \"142e9821-410f-44b3-9366-6e7f6419ac79\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:42:02.572121 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.572089 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:02.665019 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.664322 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" Apr 24 16:42:02.696541 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.696504 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c6f5d86d4-87vd7"] Apr 24 16:42:02.789328 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:02.789292 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b66498_b5b4_403e_8f0c_6754367fa215.slice/crio-1dc6e093d168b4d48a5d6f9b4e33aafa81d8b67c4d69c1c1362ff6cc7bc240da WatchSource:0}: Error finding container 1dc6e093d168b4d48a5d6f9b4e33aafa81d8b67c4d69c1c1362ff6cc7bc240da: Status 404 returned error can't find the container with id 1dc6e093d168b4d48a5d6f9b4e33aafa81d8b67c4d69c1c1362ff6cc7bc240da Apr 24 16:42:02.803995 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:02.803951 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4"] Apr 24 16:42:02.807564 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:02.807537 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142e9821_410f_44b3_9366_6e7f6419ac79.slice/crio-5c231631f41719d816ade3cf881aace6cd7112d10ed2914f22a8fdf0805392b3 WatchSource:0}: Error finding container 5c231631f41719d816ade3cf881aace6cd7112d10ed2914f22a8fdf0805392b3: Status 404 returned error can't find the container with id 5c231631f41719d816ade3cf881aace6cd7112d10ed2914f22a8fdf0805392b3 Apr 24 16:42:03.311752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.311645 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xlfzt" event={"ID":"9d427b7d-8313-4e70-8dd1-f8ae2049538f","Type":"ContainerStarted","Data":"4c7e14301416171dfe3804ebba23c142391cef096058b7cf634d3d2672f78e36"} Apr 24 16:42:03.312783 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.312755 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" event={"ID":"142e9821-410f-44b3-9366-6e7f6419ac79","Type":"ContainerStarted","Data":"5c231631f41719d816ade3cf881aace6cd7112d10ed2914f22a8fdf0805392b3"} Apr 24 16:42:03.314213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.314182 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" event={"ID":"15b66498-b5b4-403e-8f0c-6754367fa215","Type":"ContainerStarted","Data":"3145c587d51de0df9836ed6e62c9576066e80a619a7171235e0af6c00d5721d6"} Apr 24 16:42:03.314346 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.314220 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" event={"ID":"15b66498-b5b4-403e-8f0c-6754367fa215","Type":"ContainerStarted","Data":"1dc6e093d168b4d48a5d6f9b4e33aafa81d8b67c4d69c1c1362ff6cc7bc240da"} Apr 24 16:42:03.335148 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.334969 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" podStartSLOduration=33.334954081 podStartE2EDuration="33.334954081s" podCreationTimestamp="2026-04-24 16:41:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:42:03.333535053 +0000 UTC m=+164.082705839" watchObservedRunningTime="2026-04-24 16:42:03.334954081 +0000 UTC m=+164.084124845" Apr 24 16:42:03.572610 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.572521 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:03.575770 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:03.575739 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:04.318246 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.318208 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xlfzt" event={"ID":"9d427b7d-8313-4e70-8dd1-f8ae2049538f","Type":"ContainerStarted","Data":"0871c71a9a714528dc218e742dd6482c23fbbb86d1820baa567879c7837033c4"} Apr 24 16:42:04.319526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.319499 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" event={"ID":"142e9821-410f-44b3-9366-6e7f6419ac79","Type":"ContainerStarted","Data":"fb89507ecc002749b19fb02d43aff79deb8fc62c829f374b1888a1169a7427cd"} Apr 24 16:42:04.319647 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.319603 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:04.320878 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.320858 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7c6f5d86d4-87vd7" Apr 24 16:42:04.336311 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.336274 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xlfzt" podStartSLOduration=1.335253577 podStartE2EDuration="3.336260213s" podCreationTimestamp="2026-04-24 16:42:01 +0000 UTC" firstStartedPulling="2026-04-24 16:42:02.22684708 +0000 UTC m=+162.976017823" lastFinishedPulling="2026-04-24 16:42:04.227853719 +0000 UTC m=+164.977024459" observedRunningTime="2026-04-24 16:42:04.33600984 +0000 UTC m=+165.085180600" watchObservedRunningTime="2026-04-24 16:42:04.336260213 +0000 UTC m=+165.085430975" Apr 24 16:42:04.350359 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.350322 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-cb95c66f6-6c8j4" podStartSLOduration=33.516480821 podStartE2EDuration="34.350310507s" podCreationTimestamp="2026-04-24 16:41:30 +0000 UTC" firstStartedPulling="2026-04-24 16:42:02.834877756 +0000 UTC m=+163.584048496" lastFinishedPulling="2026-04-24 16:42:03.668707438 +0000 UTC m=+164.417878182" observedRunningTime="2026-04-24 16:42:04.349894763 +0000 UTC m=+165.099065525" watchObservedRunningTime="2026-04-24 16:42:04.350310507 +0000 UTC m=+165.099481267" Apr 24 16:42:04.950403 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.950371 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99"] Apr 24 16:42:04.953455 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.953439 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:04.955737 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.955719 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9txtz\"" Apr 24 16:42:04.955941 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.955929 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Apr 24 16:42:04.961612 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:04.961588 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99"] Apr 24 16:42:05.040105 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:05.040075 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cfc197c-9833-4185-8321-b5df76b2eea1-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-9kc99\" (UID: \"7cfc197c-9833-4185-8321-b5df76b2eea1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:05.141449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:05.141420 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cfc197c-9833-4185-8321-b5df76b2eea1-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-9kc99\" (UID: \"7cfc197c-9833-4185-8321-b5df76b2eea1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:05.143590 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:05.143571 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7cfc197c-9833-4185-8321-b5df76b2eea1-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-9kc99\" (UID: \"7cfc197c-9833-4185-8321-b5df76b2eea1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:05.263475 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:05.263400 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:05.374311 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:05.374283 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99"] Apr 24 16:42:05.377580 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:05.377552 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfc197c_9833_4185_8321_b5df76b2eea1.slice/crio-7301eae4976e3a5b292ca87615e565d4ced421e192a5a1914a1b8d535361d4a0 WatchSource:0}: Error finding container 7301eae4976e3a5b292ca87615e565d4ced421e192a5a1914a1b8d535361d4a0: Status 404 returned error can't find the container with id 7301eae4976e3a5b292ca87615e565d4ced421e192a5a1914a1b8d535361d4a0 Apr 24 16:42:06.327054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:06.327025 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" event={"ID":"7cfc197c-9833-4185-8321-b5df76b2eea1","Type":"ContainerStarted","Data":"7301eae4976e3a5b292ca87615e565d4ced421e192a5a1914a1b8d535361d4a0"} Apr 24 16:42:07.331600 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:07.331567 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" event={"ID":"7cfc197c-9833-4185-8321-b5df76b2eea1","Type":"ContainerStarted","Data":"902fd0d38431db4320c59c6850be13e161cf0708328a160b7026ca680bdc8b55"} Apr 24 16:42:07.332005 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:07.331767 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:07.336274 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:07.336253 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" Apr 24 16:42:07.347424 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:07.347364 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-9kc99" podStartSLOduration=2.439042826 podStartE2EDuration="3.347348943s" podCreationTimestamp="2026-04-24 16:42:04 +0000 UTC" firstStartedPulling="2026-04-24 16:42:05.379338101 +0000 UTC m=+166.128508843" lastFinishedPulling="2026-04-24 16:42:06.287644216 +0000 UTC m=+167.036814960" observedRunningTime="2026-04-24 16:42:07.347169248 +0000 UTC m=+168.096340011" watchObservedRunningTime="2026-04-24 16:42:07.347348943 +0000 UTC m=+168.096519704" Apr 24 16:42:08.845932 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:08.845898 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:42:11.488687 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:11.488641 2561 patch_prober.go:28] interesting pod/image-registry-558d8b869f-nzxwd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Apr 24 16:42:11.489157 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:11.488728 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Apr 24 16:42:12.317054 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:12.317019 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jqv7x" Apr 24 16:42:13.378983 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.378951 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-r4f2j"] Apr 24 16:42:13.383721 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.383703 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.386286 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.386263 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Apr 24 16:42:13.386389 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.386272 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Apr 24 16:42:13.386604 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.386587 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 24 16:42:13.386648 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.386587 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cfzp7\"" Apr 24 16:42:13.387256 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.387240 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 24 16:42:13.387613 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.387594 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Apr 24 16:42:13.387721 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.387597 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 24 16:42:13.389862 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.389839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.389943 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.389880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.389997 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.389968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.390044 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.390009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5k4m\" (UniqueName: \"kubernetes.io/projected/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-api-access-p5k4m\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.390344 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.390038 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.390344 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.390087 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.393448 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.393428 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-r4f2j"] Apr 24 16:42:13.408014 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.407991 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4r9fk"] Apr 24 16:42:13.410944 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.410929 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.413283 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.413264 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 24 16:42:13.413365 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.413270 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 24 16:42:13.413365 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.413323 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 24 16:42:13.413482 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.413427 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4464q\"" Apr 24 16:42:13.491367 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-wtmp\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491520 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491373 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491520 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491398 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.491520 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491520 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491476 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491522 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-metrics-client-ca\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-tls\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:13.491578 2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-root\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:13.491658 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls podName:10734571-7a9a-4ec7-b1bd-53f3f501d9f3 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:13.99163556 +0000 UTC m=+174.740806315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls") pod "kube-state-metrics-69db897b98-r4f2j" (UID: "10734571-7a9a-4ec7-b1bd-53f3f501d9f3") : secret "kube-state-metrics-tls" not found Apr 24 16:42:13.491746 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491717 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-textfile\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491757 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491805 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m529q\" (UniqueName: \"kubernetes.io/projected/47e1def0-fc20-457c-b76c-01fba0d99c84-kube-api-access-m529q\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491844 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491857 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5k4m\" (UniqueName: \"kubernetes.io/projected/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-api-access-p5k4m\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491899 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491952 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492038 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.491986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-sys\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.492556 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.492539 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.492923 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.492901 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.494079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.494063 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.500152 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.500133 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5k4m\" (UniqueName: \"kubernetes.io/projected/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-api-access-p5k4m\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.592583 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592559 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m529q\" (UniqueName: \"kubernetes.io/projected/47e1def0-fc20-457c-b76c-01fba0d99c84-kube-api-access-m529q\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592728 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592602 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-sys\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592728 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592627 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-wtmp\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592728 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592653 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592728 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592686 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-sys\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592728 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592693 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592742 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-metrics-client-ca\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592764 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-tls\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592809 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-root\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592852 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-wtmp\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592860 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-textfile\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.592971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.592888 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e1def0-fc20-457c-b76c-01fba0d99c84-root\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.593167 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.593136 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-textfile\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.593371 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.593354 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-metrics-client-ca\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.593414 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.593368 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-accelerators-collector-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.594854 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.594837 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.594947 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.594933 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e1def0-fc20-457c-b76c-01fba0d99c84-node-exporter-tls\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.601412 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.601391 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m529q\" (UniqueName: \"kubernetes.io/projected/47e1def0-fc20-457c-b76c-01fba0d99c84-kube-api-access-m529q\") pod \"node-exporter-4r9fk\" (UID: \"47e1def0-fc20-457c-b76c-01fba0d99c84\") " pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.719125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.719049 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4r9fk" Apr 24 16:42:13.726614 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:13.726588 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e1def0_fc20_457c_b76c_01fba0d99c84.slice/crio-87933bd3a5d99e9e6b6800574b69f62833af9bc6c1e97161807ef11f54f6b9a1 WatchSource:0}: Error finding container 87933bd3a5d99e9e6b6800574b69f62833af9bc6c1e97161807ef11f54f6b9a1: Status 404 returned error can't find the container with id 87933bd3a5d99e9e6b6800574b69f62833af9bc6c1e97161807ef11f54f6b9a1 Apr 24 16:42:13.997238 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.997153 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:13.999416 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:13.999391 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10734571-7a9a-4ec7-b1bd-53f3f501d9f3-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-r4f2j\" (UID: \"10734571-7a9a-4ec7-b1bd-53f3f501d9f3\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:14.293429 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.293336 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" Apr 24 16:42:14.353246 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.353211 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4r9fk" event={"ID":"47e1def0-fc20-457c-b76c-01fba0d99c84","Type":"ContainerStarted","Data":"87933bd3a5d99e9e6b6800574b69f62833af9bc6c1e97161807ef11f54f6b9a1"} Apr 24 16:42:14.429747 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.429721 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:42:14.434231 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.434211 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.436492 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.436470 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 24 16:42:14.436763 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.436748 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 24 16:42:14.436831 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.436766 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 24 16:42:14.436874 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.436766 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437098 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437122 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437171 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437176 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437189 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-925cv\"" Apr 24 16:42:14.437254 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.437197 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 24 16:42:14.445155 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.445135 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:42:14.451361 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.451340 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-r4f2j"] Apr 24 16:42:14.453897 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:14.453873 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10734571_7a9a_4ec7_b1bd_53f3f501d9f3.slice/crio-cba7cdfc8848181c9a5da2fcfa9c4b1dbbdae013d8c9f1b7ffa0bd27c60223b4 WatchSource:0}: Error finding container cba7cdfc8848181c9a5da2fcfa9c4b1dbbdae013d8c9f1b7ffa0bd27c60223b4: Status 404 returned error can't find the container with id cba7cdfc8848181c9a5da2fcfa9c4b1dbbdae013d8c9f1b7ffa0bd27c60223b4 Apr 24 16:42:14.500940 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.500904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.500940 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.500940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501132 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.500968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrsq\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501132 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501132 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501057 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501132 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501273 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501273 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501218 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501273 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501246 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501412 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501412 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501392 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501507 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501438 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.501507 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.501467 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602602 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602538 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602602 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602574 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602602 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-svrsq\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602628 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602697 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602734 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602975 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602796 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602975 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602838 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.602975 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:14.602866 2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Apr 24 16:42:14.602975 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.602868 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.603697 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:14.603618 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls podName:c88c3682-2746-4ac3-adb4-b2fc7e771350 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:15.102911664 +0000 UTC m=+175.852082414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350") : secret "alertmanager-main-tls" not found Apr 24 16:42:14.603839 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.603728 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.603938 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.603917 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.605655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.604124 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.605655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.604291 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.605655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.604456 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.605655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.604930 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.606954 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.606167 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.606954 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.606193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.606954 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:14.606573 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle podName:c88c3682-2746-4ac3-adb4-b2fc7e771350 nodeName:}" failed. No retries permitted until 2026-04-24 16:42:15.106535994 +0000 UTC m=+175.855706741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350") : configmap references non-existent config key: ca-bundle.crt Apr 24 16:42:14.606954 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.606606 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.609228 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.607430 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.609228 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.607683 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.609228 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.608631 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.609420 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.609363 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.611231 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.611208 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:14.612873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:14.612854 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrsq\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.109023 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.108987 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.109184 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.109110 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.109892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.109863 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.111654 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.111622 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.344119 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.344086 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:42:15.357717 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.357680 2561 generic.go:358] "Generic (PLEG): container finished" podID="47e1def0-fc20-457c-b76c-01fba0d99c84" containerID="c7c41043435652366bbb6edba1518cb0f8aafcdd398db51fe0dbd1c6d4a91546" exitCode=0 Apr 24 16:42:15.357856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.357770 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4r9fk" event={"ID":"47e1def0-fc20-457c-b76c-01fba0d99c84","Type":"ContainerDied","Data":"c7c41043435652366bbb6edba1518cb0f8aafcdd398db51fe0dbd1c6d4a91546"} Apr 24 16:42:15.359099 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.359020 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" event={"ID":"10734571-7a9a-4ec7-b1bd-53f3f501d9f3","Type":"ContainerStarted","Data":"cba7cdfc8848181c9a5da2fcfa9c4b1dbbdae013d8c9f1b7ffa0bd27c60223b4"} Apr 24 16:42:15.612376 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:15.612338 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:42:15.615462 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:15.615373 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88c3682_2746_4ac3_adb4_b2fc7e771350.slice/crio-ed57269836966015ee0ae534771bc87d451b8dd6a2f66737c38efb63a6b933c5 WatchSource:0}: Error finding container ed57269836966015ee0ae534771bc87d451b8dd6a2f66737c38efb63a6b933c5: Status 404 returned error can't find the container with id ed57269836966015ee0ae534771bc87d451b8dd6a2f66737c38efb63a6b933c5 Apr 24 16:42:16.364641 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.364586 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4r9fk" event={"ID":"47e1def0-fc20-457c-b76c-01fba0d99c84","Type":"ContainerStarted","Data":"6f564bb5dede7dd2c918fef972de496f959b6a99b3f905fc01879a6f827fa714"} Apr 24 16:42:16.364641 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.364634 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4r9fk" event={"ID":"47e1def0-fc20-457c-b76c-01fba0d99c84","Type":"ContainerStarted","Data":"4e5a813fc9ef11dbf15701183cf70bc6c901302534ea36cbfb61e795c4e0e4bb"} Apr 24 16:42:16.366658 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.366619 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" event={"ID":"10734571-7a9a-4ec7-b1bd-53f3f501d9f3","Type":"ContainerStarted","Data":"039f9b4b450a9bac520301d3285084bda61a0e8064f809c4fd850375afadb99f"} Apr 24 16:42:16.366781 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.366690 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" event={"ID":"10734571-7a9a-4ec7-b1bd-53f3f501d9f3","Type":"ContainerStarted","Data":"d6ac17f21dff9b36cd0aa1e5abf7b4ba84e3c305a6c5887e126c279a3845616d"} Apr 24 16:42:16.366781 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.366711 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" event={"ID":"10734571-7a9a-4ec7-b1bd-53f3f501d9f3","Type":"ContainerStarted","Data":"9878fb46ecaea560697dac21e74949da4c85cc2ae49c3e5a836199ee60e79bf9"} Apr 24 16:42:16.367768 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.367747 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"ed57269836966015ee0ae534771bc87d451b8dd6a2f66737c38efb63a6b933c5"} Apr 24 16:42:16.386944 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.386905 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4r9fk" podStartSLOduration=2.737501363 podStartE2EDuration="3.386892119s" podCreationTimestamp="2026-04-24 16:42:13 +0000 UTC" firstStartedPulling="2026-04-24 16:42:13.728141361 +0000 UTC m=+174.477312100" lastFinishedPulling="2026-04-24 16:42:14.377532103 +0000 UTC m=+175.126702856" observedRunningTime="2026-04-24 16:42:16.385160505 +0000 UTC m=+177.134331260" watchObservedRunningTime="2026-04-24 16:42:16.386892119 +0000 UTC m=+177.136062874" Apr 24 16:42:16.406128 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:16.405961 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-69db897b98-r4f2j" podStartSLOduration=2.318544085 podStartE2EDuration="3.405943315s" podCreationTimestamp="2026-04-24 16:42:13 +0000 UTC" firstStartedPulling="2026-04-24 16:42:14.455682465 +0000 UTC m=+175.204853208" lastFinishedPulling="2026-04-24 16:42:15.54308169 +0000 UTC m=+176.292252438" observedRunningTime="2026-04-24 16:42:16.405327653 +0000 UTC m=+177.154498427" watchObservedRunningTime="2026-04-24 16:42:16.405943315 +0000 UTC m=+177.155114077" Apr 24 16:42:17.372409 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:17.372360 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16" exitCode=0 Apr 24 16:42:17.372904 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:17.372458 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16"} Apr 24 16:42:19.380617 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.380587 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24"} Apr 24 16:42:19.380617 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.380618 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b"} Apr 24 16:42:19.381024 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.380628 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad"} Apr 24 16:42:19.381024 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.380637 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e"} Apr 24 16:42:19.381024 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.380645 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23"} Apr 24 16:42:19.682924 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.682849 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:42:19.686401 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.686385 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.689727 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.689703 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 24 16:42:19.689727 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.689704 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 24 16:42:19.689892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.689754 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4lcbz\"" Apr 24 16:42:19.689892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.689763 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 24 16:42:19.690511 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690487 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 24 16:42:19.690511 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690509 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 24 16:42:19.690732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690537 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 24 16:42:19.690732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690557 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 24 16:42:19.690732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690554 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 24 16:42:19.690732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690509 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 24 16:42:19.690732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.690654 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 24 16:42:19.691473 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.691458 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 24 16:42:19.691845 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.691827 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-811blobdj7vv8\"" Apr 24 16:42:19.693138 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.693119 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 24 16:42:19.696096 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.696079 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 24 16:42:19.706600 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.706579 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:42:19.751599 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751869 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751869 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751981 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.751981 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752068 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.751997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752068 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752035 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752150 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752150 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752097 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752150 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752130 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752230 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752192 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752230 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752316 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752244 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752316 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752273 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752419 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.752419 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.752340 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8879j\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852756 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852798 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852828 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852854 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852900 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852928 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.852955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852952 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.852978 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853013 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853077 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853106 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853129 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853161 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853188 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853197 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853390 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8879j\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853438 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853478 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.853633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.853507 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.854215 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.854195 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.855805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.855693 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.856406 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.856390 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 24 16:42:19.859053 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859033 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 24 16:42:19.859173 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859116 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 24 16:42:19.859253 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859192 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 24 16:42:19.859253 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859226 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 24 16:42:19.859349 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859304 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-811blobdj7vv8\"" Apr 24 16:42:19.859984 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.859968 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 24 16:42:19.860509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.860293 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 24 16:42:19.860509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.860305 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 24 16:42:19.860509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.860361 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 24 16:42:19.860509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.860360 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 24 16:42:19.860813 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.860795 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 24 16:42:19.862098 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.862081 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 24 16:42:19.865345 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.865300 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.865643 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.865615 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.867789 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.867764 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.868091 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.868067 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.868735 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.868368 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8879j\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.868735 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.868683 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.868912 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.868883 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.869633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.869588 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.870136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870093 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.870136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870117 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.870299 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870284 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 24 16:42:19.870888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870642 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.870888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870759 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.871016 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.870888 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.871699 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.871640 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.874073 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.874050 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:19.998035 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:19.998014 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4lcbz\"" Apr 24 16:42:20.006805 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.006779 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:42:20.135941 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.135912 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:42:20.139227 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:42:20.139189 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68f78b9_dc26_43dd_91ba_c06247483ca0.slice/crio-4105fdf4a5b31f9b189dde7fed94bceba7f638febf9c4b89ee1453f42226cc27 WatchSource:0}: Error finding container 4105fdf4a5b31f9b189dde7fed94bceba7f638febf9c4b89ee1453f42226cc27: Status 404 returned error can't find the container with id 4105fdf4a5b31f9b189dde7fed94bceba7f638febf9c4b89ee1453f42226cc27 Apr 24 16:42:20.385958 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.385922 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerStarted","Data":"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed"} Apr 24 16:42:20.387291 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.387263 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667" exitCode=0 Apr 24 16:42:20.387421 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.387307 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667"} Apr 24 16:42:20.387421 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.387346 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"4105fdf4a5b31f9b189dde7fed94bceba7f638febf9c4b89ee1453f42226cc27"} Apr 24 16:42:20.412983 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:20.412944 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.034912351 podStartE2EDuration="6.412931938s" podCreationTimestamp="2026-04-24 16:42:14 +0000 UTC" firstStartedPulling="2026-04-24 16:42:15.617926707 +0000 UTC m=+176.367097454" lastFinishedPulling="2026-04-24 16:42:19.995946298 +0000 UTC m=+180.745117041" observedRunningTime="2026-04-24 16:42:20.411868835 +0000 UTC m=+181.161039595" watchObservedRunningTime="2026-04-24 16:42:20.412931938 +0000 UTC m=+181.162102697" Apr 24 16:42:21.489359 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:21.489332 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:42:23.319785 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:23.319759 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8b56f5b5b-jclzv" Apr 24 16:42:23.399267 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:23.399235 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6"} Apr 24 16:42:23.399414 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:23.399276 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000"} Apr 24 16:42:25.408125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:25.408085 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570"} Apr 24 16:42:25.408125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:25.408119 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1"} Apr 24 16:42:25.408125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:25.408131 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d"} Apr 24 16:42:25.408538 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:25.408140 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerStarted","Data":"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833"} Apr 24 16:42:25.438043 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:25.437994 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.383899051 podStartE2EDuration="6.437980285s" podCreationTimestamp="2026-04-24 16:42:19 +0000 UTC" firstStartedPulling="2026-04-24 16:42:20.388317893 +0000 UTC m=+181.137488635" lastFinishedPulling="2026-04-24 16:42:24.442399129 +0000 UTC m=+185.191569869" observedRunningTime="2026-04-24 16:42:25.43553879 +0000 UTC m=+186.184709550" watchObservedRunningTime="2026-04-24 16:42:25.437980285 +0000 UTC m=+186.187151046" Apr 24 16:42:26.508446 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.508388 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" containerID="cri-o://6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689" gracePeriod=30 Apr 24 16:42:26.738462 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.738441 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:42:26.817692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817614 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817653 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817684 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5g7s\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817719 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817747 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817796 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817851 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.817893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.817882 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration\") pod \"72164181-861f-46e6-9c99-34e40609fa17\" (UID: \"72164181-861f-46e6-9c99-34e40609fa17\") " Apr 24 16:42:26.818255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.818226 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:42:26.818374 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.818264 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:42:26.820191 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.820157 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:42:26.820306 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.820293 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s" (OuterVolumeSpecName: "kube-api-access-l5g7s") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "kube-api-access-l5g7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:42:26.820371 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.820327 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:42:26.820422 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.820406 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:42:26.820489 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.820474 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:42:26.826679 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.826646 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "72164181-861f-46e6-9c99-34e40609fa17" (UID: "72164181-861f-46e6-9c99-34e40609fa17"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:42:26.918688 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918616 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-registry-certificates\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918688 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918691 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-installation-pull-secrets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918708 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72164181-861f-46e6-9c99-34e40609fa17-image-registry-private-configuration\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918722 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-registry-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918738 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72164181-861f-46e6-9c99-34e40609fa17-ca-trust-extracted\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918749 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5g7s\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-kube-api-access-l5g7s\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918762 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72164181-861f-46e6-9c99-34e40609fa17-bound-sa-token\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:26.918860 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:26.918774 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72164181-861f-46e6-9c99-34e40609fa17-trusted-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:42:27.414616 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.414586 2561 generic.go:358] "Generic (PLEG): container finished" podID="72164181-861f-46e6-9c99-34e40609fa17" containerID="6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689" exitCode=0 Apr 24 16:42:27.414777 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.414643 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" event={"ID":"72164181-861f-46e6-9c99-34e40609fa17","Type":"ContainerDied","Data":"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689"} Apr 24 16:42:27.414777 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.414647 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" Apr 24 16:42:27.414777 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.414686 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d8b869f-nzxwd" event={"ID":"72164181-861f-46e6-9c99-34e40609fa17","Type":"ContainerDied","Data":"45d0a44cde771a40ec683cf1543bf598b566497673330ff6afca4ecf5c2a7936"} Apr 24 16:42:27.414777 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.414703 2561 scope.go:117] "RemoveContainer" containerID="6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689" Apr 24 16:42:27.423661 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.423645 2561 scope.go:117] "RemoveContainer" containerID="6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689" Apr 24 16:42:27.423939 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:42:27.423923 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689\": container with ID starting with 6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689 not found: ID does not exist" containerID="6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689" Apr 24 16:42:27.423977 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.423946 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689"} err="failed to get container status \"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689\": rpc error: code = NotFound desc = could not find container \"6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689\": container with ID starting with 6c44c4b67d4d851df7e97daa2d40e614b795608d886f14f7ed576f03089b7689 not found: ID does not exist" Apr 24 16:42:27.436891 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.436866 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:42:27.440325 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.440304 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-558d8b869f-nzxwd"] Apr 24 16:42:27.849448 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:27.849373 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72164181-861f-46e6-9c99-34e40609fa17" path="/var/lib/kubelet/pods/72164181-861f-46e6-9c99-34e40609fa17/volumes" Apr 24 16:42:30.007127 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:42:30.007084 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:07.153821 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.153789 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/init-config-reloader/0.log" Apr 24 16:43:07.163012 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.162991 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/alertmanager/0.log" Apr 24 16:43:07.296134 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.296102 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/config-reloader/0.log" Apr 24 16:43:07.498378 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.498290 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/kube-rbac-proxy-web/0.log" Apr 24 16:43:07.696263 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.696231 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/kube-rbac-proxy/0.log" Apr 24 16:43:07.896465 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:07.896427 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/kube-rbac-proxy-metric/0.log" Apr 24 16:43:08.098601 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:08.098577 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c88c3682-2746-4ac3-adb4-b2fc7e771350/prom-label-proxy/0.log" Apr 24 16:43:08.496036 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:08.496011 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-state-metrics/0.log" Apr 24 16:43:08.695891 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:08.695860 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-rbac-proxy-main/0.log" Apr 24 16:43:08.895859 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:08.895826 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-rbac-proxy-self/0.log" Apr 24 16:43:10.094845 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:10.094820 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/init-textfile/0.log" Apr 24 16:43:10.296343 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:10.296316 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/node-exporter/0.log" Apr 24 16:43:10.495799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:10.495772 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/kube-rbac-proxy/0.log" Apr 24 16:43:11.895732 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:11.895702 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/init-config-reloader/0.log" Apr 24 16:43:12.096811 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:12.096781 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/prometheus/0.log" Apr 24 16:43:12.296299 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:12.296226 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/config-reloader/0.log" Apr 24 16:43:12.495900 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:12.495876 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/thanos-sidecar/0.log" Apr 24 16:43:12.695414 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:12.695391 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/kube-rbac-proxy-web/0.log" Apr 24 16:43:12.895893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:12.895867 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/kube-rbac-proxy/0.log" Apr 24 16:43:13.096723 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:13.096631 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e68f78b9-dc26-43dd-91ba-c06247483ca0/kube-rbac-proxy-thanos/0.log" Apr 24 16:43:13.695854 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:13.695821 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-9kc99_7cfc197c-9833-4185-8321-b5df76b2eea1/prometheus-operator-admission-webhook/0.log" Apr 24 16:43:15.095655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:15.095622 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-cb95c66f6-6c8j4_142e9821-410f-44b3-9366-6e7f6419ac79/networking-console-plugin/0.log" Apr 24 16:43:20.007562 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:20.007528 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:20.024540 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:20.024464 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:20.605710 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:20.605680 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:31.567503 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:31.567461 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:43:31.569863 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:31.569837 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71bdf81-e127-49a6-aca6-ff4846028fd9-metrics-certs\") pod \"network-metrics-daemon-mwzw9\" (UID: \"b71bdf81-e127-49a6-aca6-ff4846028fd9\") " pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:43:31.649509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:31.649475 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-czswq\"" Apr 24 16:43:31.657470 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:31.657451 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwzw9" Apr 24 16:43:31.771326 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:31.771288 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwzw9"] Apr 24 16:43:31.774612 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:43:31.774583 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71bdf81_e127_49a6_aca6_ff4846028fd9.slice/crio-8f6a8288e71246a35ceac922f5cc0c41d8d676feb5aadb6ec05b5b3a5a25d9fa WatchSource:0}: Error finding container 8f6a8288e71246a35ceac922f5cc0c41d8d676feb5aadb6ec05b5b3a5a25d9fa: Status 404 returned error can't find the container with id 8f6a8288e71246a35ceac922f5cc0c41d8d676feb5aadb6ec05b5b3a5a25d9fa Apr 24 16:43:32.619087 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:32.619016 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzw9" event={"ID":"b71bdf81-e127-49a6-aca6-ff4846028fd9","Type":"ContainerStarted","Data":"8f6a8288e71246a35ceac922f5cc0c41d8d676feb5aadb6ec05b5b3a5a25d9fa"} Apr 24 16:43:33.623526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.623483 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzw9" event={"ID":"b71bdf81-e127-49a6-aca6-ff4846028fd9","Type":"ContainerStarted","Data":"f688b2d8a0363b9798fd3bacaaa61cb5226c71840fae21ccea0ef579bf39c548"} Apr 24 16:43:33.623526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.623525 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwzw9" event={"ID":"b71bdf81-e127-49a6-aca6-ff4846028fd9","Type":"ContainerStarted","Data":"17476fab3d19570b102149f722d02c7c7771f621c373c455e36ef2aff67ec430"} Apr 24 16:43:33.640797 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.640283 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mwzw9" podStartSLOduration=252.778226328 podStartE2EDuration="4m13.640264813s" podCreationTimestamp="2026-04-24 16:39:20 +0000 UTC" firstStartedPulling="2026-04-24 16:43:31.776558171 +0000 UTC m=+252.525728910" lastFinishedPulling="2026-04-24 16:43:32.638596646 +0000 UTC m=+253.387767395" observedRunningTime="2026-04-24 16:43:33.638758559 +0000 UTC m=+254.387929320" watchObservedRunningTime="2026-04-24 16:43:33.640264813 +0000 UTC m=+254.389435576" Apr 24 16:43:33.721573 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.721539 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:33.722004 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.721981 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="alertmanager" containerID="cri-o://fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" gracePeriod=120 Apr 24 16:43:33.722076 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.722050 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-metric" containerID="cri-o://4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" gracePeriod=120 Apr 24 16:43:33.722142 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.722052 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-web" containerID="cri-o://298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" gracePeriod=120 Apr 24 16:43:33.722142 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.722094 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="config-reloader" containerID="cri-o://01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" gracePeriod=120 Apr 24 16:43:33.722247 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.722140 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="prom-label-proxy" containerID="cri-o://c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" gracePeriod=120 Apr 24 16:43:33.722247 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:33.722073 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy" containerID="cri-o://19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" gracePeriod=120 Apr 24 16:43:34.629751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629720 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" exitCode=0 Apr 24 16:43:34.629751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629744 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" exitCode=0 Apr 24 16:43:34.629751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629750 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" exitCode=0 Apr 24 16:43:34.629751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629757 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" exitCode=0 Apr 24 16:43:34.630206 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629790 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed"} Apr 24 16:43:34.630206 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629825 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b"} Apr 24 16:43:34.630206 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629836 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e"} Apr 24 16:43:34.630206 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.629845 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23"} Apr 24 16:43:34.964742 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:34.964717 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.097333 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097294 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097333 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097330 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097527 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097361 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097527 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097387 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097527 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097516 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097554 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svrsq\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097581 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097602 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097651 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097708 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097751 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097759 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097774 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097779 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:35.097911 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.097803 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c88c3682-2746-4ac3-adb4-b2fc7e771350\" (UID: \"c88c3682-2746-4ac3-adb4-b2fc7e771350\") " Apr 24 16:43:35.098258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.098048 2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.098258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.098068 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c88c3682-2746-4ac3-adb4-b2fc7e771350-metrics-client-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.098258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.098047 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:43:35.100142 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100107 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.100244 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100175 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.100313 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100237 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:43:35.100644 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100615 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out" (OuterVolumeSpecName: "config-out") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:43:35.100935 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100901 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.101045 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.100993 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume" (OuterVolumeSpecName: "config-volume") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.101342 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.101316 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq" (OuterVolumeSpecName: "kube-api-access-svrsq") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "kube-api-access-svrsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:43:35.101572 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.101557 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.104578 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.104553 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.110824 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.110800 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config" (OuterVolumeSpecName: "web-config") pod "c88c3682-2746-4ac3-adb4-b2fc7e771350" (UID: "c88c3682-2746-4ac3-adb4-b2fc7e771350"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198448 2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-alertmanager-main-db\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198482 2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198492 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-web-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198500 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198511 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198521 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-main-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198526 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198529 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-tls-assets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198844 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198542 2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198844 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198555 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svrsq\" (UniqueName: \"kubernetes.io/projected/c88c3682-2746-4ac3-adb4-b2fc7e771350-kube-api-access-svrsq\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198844 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198563 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c88c3682-2746-4ac3-adb4-b2fc7e771350-config-out\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.198844 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.198572 2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c88c3682-2746-4ac3-adb4-b2fc7e771350-cluster-tls-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:35.636033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636004 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" exitCode=0 Apr 24 16:43:35.636033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636028 2561 generic.go:358] "Generic (PLEG): container finished" podID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerID="298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" exitCode=0 Apr 24 16:43:35.636449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636075 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24"} Apr 24 16:43:35.636449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636102 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad"} Apr 24 16:43:35.636449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636114 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c88c3682-2746-4ac3-adb4-b2fc7e771350","Type":"ContainerDied","Data":"ed57269836966015ee0ae534771bc87d451b8dd6a2f66737c38efb63a6b933c5"} Apr 24 16:43:35.636449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636112 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.636449 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.636125 2561 scope.go:117] "RemoveContainer" containerID="c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" Apr 24 16:43:35.643591 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.643570 2561 scope.go:117] "RemoveContainer" containerID="4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" Apr 24 16:43:35.650405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.650390 2561 scope.go:117] "RemoveContainer" containerID="19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" Apr 24 16:43:35.656368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.656352 2561 scope.go:117] "RemoveContainer" containerID="298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" Apr 24 16:43:35.660079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.660035 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:35.664854 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.664793 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:35.664919 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.664893 2561 scope.go:117] "RemoveContainer" containerID="01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" Apr 24 16:43:35.671089 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.671068 2561 scope.go:117] "RemoveContainer" containerID="fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" Apr 24 16:43:35.677275 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.677258 2561 scope.go:117] "RemoveContainer" containerID="1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16" Apr 24 16:43:35.683452 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.683435 2561 scope.go:117] "RemoveContainer" containerID="c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" Apr 24 16:43:35.683714 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.683687 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed\": container with ID starting with c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed not found: ID does not exist" containerID="c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" Apr 24 16:43:35.683796 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.683720 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed"} err="failed to get container status \"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed\": rpc error: code = NotFound desc = could not find container \"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed\": container with ID starting with c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed not found: ID does not exist" Apr 24 16:43:35.683796 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.683745 2561 scope.go:117] "RemoveContainer" containerID="4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" Apr 24 16:43:35.683989 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.683974 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24\": container with ID starting with 4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24 not found: ID does not exist" containerID="4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" Apr 24 16:43:35.684023 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.683993 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24"} err="failed to get container status \"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24\": rpc error: code = NotFound desc = could not find container \"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24\": container with ID starting with 4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24 not found: ID does not exist" Apr 24 16:43:35.684023 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684006 2561 scope.go:117] "RemoveContainer" containerID="19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" Apr 24 16:43:35.684229 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.684215 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b\": container with ID starting with 19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b not found: ID does not exist" containerID="19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" Apr 24 16:43:35.684288 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684231 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b"} err="failed to get container status \"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b\": rpc error: code = NotFound desc = could not find container \"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b\": container with ID starting with 19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b not found: ID does not exist" Apr 24 16:43:35.684288 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684246 2561 scope.go:117] "RemoveContainer" containerID="298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" Apr 24 16:43:35.684453 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.684435 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad\": container with ID starting with 298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad not found: ID does not exist" containerID="298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" Apr 24 16:43:35.684493 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684459 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad"} err="failed to get container status \"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad\": rpc error: code = NotFound desc = could not find container \"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad\": container with ID starting with 298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad not found: ID does not exist" Apr 24 16:43:35.684493 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684485 2561 scope.go:117] "RemoveContainer" containerID="01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" Apr 24 16:43:35.684726 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.684711 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e\": container with ID starting with 01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e not found: ID does not exist" containerID="01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" Apr 24 16:43:35.684770 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684730 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e"} err="failed to get container status \"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e\": rpc error: code = NotFound desc = could not find container \"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e\": container with ID starting with 01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e not found: ID does not exist" Apr 24 16:43:35.684770 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684742 2561 scope.go:117] "RemoveContainer" containerID="fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" Apr 24 16:43:35.684967 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.684953 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23\": container with ID starting with fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23 not found: ID does not exist" containerID="fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" Apr 24 16:43:35.685001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684970 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23"} err="failed to get container status \"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23\": rpc error: code = NotFound desc = could not find container \"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23\": container with ID starting with fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23 not found: ID does not exist" Apr 24 16:43:35.685001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.684984 2561 scope.go:117] "RemoveContainer" containerID="1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16" Apr 24 16:43:35.685196 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:35.685180 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16\": container with ID starting with 1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16 not found: ID does not exist" containerID="1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16" Apr 24 16:43:35.685235 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685200 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16"} err="failed to get container status \"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16\": rpc error: code = NotFound desc = could not find container \"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16\": container with ID starting with 1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16 not found: ID does not exist" Apr 24 16:43:35.685235 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685214 2561 scope.go:117] "RemoveContainer" containerID="c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed" Apr 24 16:43:35.685426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685410 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed"} err="failed to get container status \"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed\": rpc error: code = NotFound desc = could not find container \"c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed\": container with ID starting with c895660933f5e4c6de68c9882b5d64a311bff4ef6cda0488ada73302192c96ed not found: ID does not exist" Apr 24 16:43:35.685426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685425 2561 scope.go:117] "RemoveContainer" containerID="4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24" Apr 24 16:43:35.685620 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685604 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24"} err="failed to get container status \"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24\": rpc error: code = NotFound desc = could not find container \"4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24\": container with ID starting with 4bfef567ea1acda7e1280e3c0f98da40c30cfdfc1899e98b6370cba829154a24 not found: ID does not exist" Apr 24 16:43:35.685662 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685620 2561 scope.go:117] "RemoveContainer" containerID="19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b" Apr 24 16:43:35.685823 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685807 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b"} err="failed to get container status \"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b\": rpc error: code = NotFound desc = could not find container \"19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b\": container with ID starting with 19ca691bae1872a26af94f21d8a4cfdc358d0feee9153c4c1ad3fa9044de054b not found: ID does not exist" Apr 24 16:43:35.685866 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.685824 2561 scope.go:117] "RemoveContainer" containerID="298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad" Apr 24 16:43:35.686035 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686016 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad"} err="failed to get container status \"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad\": rpc error: code = NotFound desc = could not find container \"298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad\": container with ID starting with 298eab803196929885428494a4bf142cc7126930a8a59a76a43e012deff6a8ad not found: ID does not exist" Apr 24 16:43:35.686080 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686036 2561 scope.go:117] "RemoveContainer" containerID="01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e" Apr 24 16:43:35.686269 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686252 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e"} err="failed to get container status \"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e\": rpc error: code = NotFound desc = could not find container \"01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e\": container with ID starting with 01aeb517527ee3ae10a59750c4882549a504bff9f566d718204fdd7f47f3113e not found: ID does not exist" Apr 24 16:43:35.686318 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686272 2561 scope.go:117] "RemoveContainer" containerID="fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23" Apr 24 16:43:35.686519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686498 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23"} err="failed to get container status \"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23\": rpc error: code = NotFound desc = could not find container \"fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23\": container with ID starting with fb87981798eaeb7e20719659f671893a7a8e890f232149908e34830fa1361c23 not found: ID does not exist" Apr 24 16:43:35.686519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686519 2561 scope.go:117] "RemoveContainer" containerID="1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16" Apr 24 16:43:35.686975 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.686862 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16"} err="failed to get container status \"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16\": rpc error: code = NotFound desc = could not find container \"1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16\": container with ID starting with 1c2c2705f08f4c9c74df9396d4f7b58b23ee8c26704cbc18d03fa92db7fbae16 not found: ID does not exist" Apr 24 16:43:35.688118 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688101 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:35.688385 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688374 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy" Apr 24 16:43:35.688425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688399 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy" Apr 24 16:43:35.688425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688409 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="alertmanager" Apr 24 16:43:35.688425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688415 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="alertmanager" Apr 24 16:43:35.688425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688422 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="config-reloader" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688427 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="config-reloader" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688438 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="init-config-reloader" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688443 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="init-config-reloader" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688451 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-web" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688455 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-web" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688463 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688468 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688474 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-metric" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688479 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-metric" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688485 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="prom-label-proxy" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688489 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="prom-label-proxy" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688529 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-metric" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688537 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="prom-label-proxy" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688545 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="72164181-861f-46e6-9c99-34e40609fa17" containerName="registry" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688551 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy-web" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688558 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="config-reloader" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688563 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="alertmanager" Apr 24 16:43:35.688575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.688569 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" containerName="kube-rbac-proxy" Apr 24 16:43:35.693770 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.693754 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.696217 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696195 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 24 16:43:35.696323 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696202 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 24 16:43:35.696323 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696218 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 24 16:43:35.696434 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696222 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 24 16:43:35.696554 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696538 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 24 16:43:35.696554 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696550 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 24 16:43:35.696698 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696572 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 24 16:43:35.696698 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696629 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 24 16:43:35.696698 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.696629 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-925cv\"" Apr 24 16:43:35.700956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.700940 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 24 16:43:35.704196 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.704175 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:35.801918 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.801889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.801928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-web-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.801956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-out\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.801974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.801994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-volume\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltd9\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-kube-api-access-pltd9\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802247 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802285 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.802461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.802348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.850961 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.850932 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88c3682-2746-4ac3-adb4-b2fc7e771350" path="/var/lib/kubelet/pods/c88c3682-2746-4ac3-adb4-b2fc7e771350/volumes" Apr 24 16:43:35.902803 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.902740 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.902803 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.902772 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-web-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.902803 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.902796 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-out\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.902976 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.902810 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903013 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.902989 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-volume\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903055 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903034 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903106 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903071 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903160 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903110 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903160 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903142 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pltd9\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-kube-api-access-pltd9\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903181 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903213 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903258 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903239 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903396 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903286 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.903602 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903574 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.904069 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.903874 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.904163 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.904141 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907fcbea-0f80-44fd-8371-f1a7f88e328f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.905554 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.905527 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-out\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.905839 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.905814 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.906134 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.906087 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-config-volume\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.906246 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.906178 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.906334 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.906311 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.906334 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.906325 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-web-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.906799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.906769 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.907094 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.907077 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.907418 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.907402 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/907fcbea-0f80-44fd-8371-f1a7f88e328f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:35.911636 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:35.911616 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltd9\" (UniqueName: \"kubernetes.io/projected/907fcbea-0f80-44fd-8371-f1a7f88e328f-kube-api-access-pltd9\") pod \"alertmanager-main-0\" (UID: \"907fcbea-0f80-44fd-8371-f1a7f88e328f\") " pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:36.002933 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:36.002898 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 24 16:43:36.130292 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:36.130258 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 24 16:43:36.132934 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:43:36.132908 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907fcbea_0f80_44fd_8371_f1a7f88e328f.slice/crio-d84e9222d0713d17691f4622a0bbc104529ba9bdee8c866fe2c14f2c0cbaca25 WatchSource:0}: Error finding container d84e9222d0713d17691f4622a0bbc104529ba9bdee8c866fe2c14f2c0cbaca25: Status 404 returned error can't find the container with id d84e9222d0713d17691f4622a0bbc104529ba9bdee8c866fe2c14f2c0cbaca25 Apr 24 16:43:36.640391 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:36.640358 2561 generic.go:358] "Generic (PLEG): container finished" podID="907fcbea-0f80-44fd-8371-f1a7f88e328f" containerID="a44e2d6facc8de48d667227e7d4cabf2b96c3f77ced73c38fe044983e0ef8f48" exitCode=0 Apr 24 16:43:36.640857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:36.640449 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerDied","Data":"a44e2d6facc8de48d667227e7d4cabf2b96c3f77ced73c38fe044983e0ef8f48"} Apr 24 16:43:36.640857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:36.640497 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"d84e9222d0713d17691f4622a0bbc104529ba9bdee8c866fe2c14f2c0cbaca25"} Apr 24 16:43:37.647873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647838 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"d3db18ab6df32d883203913613651cd47ca09a7aaa06a39a12e70263a5908002"} Apr 24 16:43:37.647873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647877 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"2e7d1edff2dde1ff102a15a6bd1872175cfc83427cc29f14bc7ee16e6f15a245"} Apr 24 16:43:37.648297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647891 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"cb9e9c320e5e6e95af5cdd008c29ebeffbfc5451f3e93937b87698eba68956d2"} Apr 24 16:43:37.648297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647903 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"46326bcaf897f93cdb5b7c55b343aec1466e646a6a1b99a4c4fcf79fd24dcba2"} Apr 24 16:43:37.648297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647914 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"0eb48c652a0e7b8c7216a4c265d243ba34804f48a3654a32c06396ba2ded7db4"} Apr 24 16:43:37.648297 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.647925 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"907fcbea-0f80-44fd-8371-f1a7f88e328f","Type":"ContainerStarted","Data":"bf8587f380d482c56e5b4b1c799c64930357105b44eaa34e912c02b8534a51b5"} Apr 24 16:43:37.678142 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.678090 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.678076229 podStartE2EDuration="2.678076229s" podCreationTimestamp="2026-04-24 16:43:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:43:37.676909129 +0000 UTC m=+258.426079888" watchObservedRunningTime="2026-04-24 16:43:37.678076229 +0000 UTC m=+258.427247024" Apr 24 16:43:37.785969 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.785931 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-857678c95-pmn4r"] Apr 24 16:43:37.789439 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.789420 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.793299 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793280 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Apr 24 16:43:37.793405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793288 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Apr 24 16:43:37.793405 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793326 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-l78hx\"" Apr 24 16:43:37.793705 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793689 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Apr 24 16:43:37.793751 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793711 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Apr 24 16:43:37.794006 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.793990 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Apr 24 16:43:37.801554 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.801529 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Apr 24 16:43:37.801831 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.801810 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-857678c95-pmn4r"] Apr 24 16:43:37.920883 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.920804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.920883 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.920856 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-metrics-client-ca\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921071 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.920930 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921071 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.920974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921071 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.920997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-federate-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921071 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.921026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6hj\" (UniqueName: \"kubernetes.io/projected/b09e332a-bc47-445b-ba0f-8b6740d2419b-kube-api-access-qp6hj\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921261 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.921144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-serving-certs-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:37.921261 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:37.921179 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.021912 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.021874 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.021912 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.021913 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-federate-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.021939 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6hj\" (UniqueName: \"kubernetes.io/projected/b09e332a-bc47-445b-ba0f-8b6740d2419b-kube-api-access-qp6hj\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.021972 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-serving-certs-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.022008 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.022040 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.022078 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-metrics-client-ca\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.022122 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.022930 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.022897 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-metrics-client-ca\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.023067 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.023046 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-serving-certs-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.023140 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.023084 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.024689 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.024641 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-federate-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.024778 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.024646 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-telemeter-client-tls\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.024831 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.024805 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.024974 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.024950 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b09e332a-bc47-445b-ba0f-8b6740d2419b-secret-telemeter-client\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.037555 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.037528 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6hj\" (UniqueName: \"kubernetes.io/projected/b09e332a-bc47-445b-ba0f-8b6740d2419b-kube-api-access-qp6hj\") pod \"telemeter-client-857678c95-pmn4r\" (UID: \"b09e332a-bc47-445b-ba0f-8b6740d2419b\") " pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.099337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.099308 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" Apr 24 16:43:38.155532 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.155500 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:38.155975 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.155942 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="prometheus" containerID="cri-o://ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000" gracePeriod=600 Apr 24 16:43:38.156109 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.155976 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy" containerID="cri-o://0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1" gracePeriod=600 Apr 24 16:43:38.156109 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.156015 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570" gracePeriod=600 Apr 24 16:43:38.156109 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.156014 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-web" containerID="cri-o://f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d" gracePeriod=600 Apr 24 16:43:38.156109 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.156074 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="config-reloader" containerID="cri-o://e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6" gracePeriod=600 Apr 24 16:43:38.156288 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.155972 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="thanos-sidecar" containerID="cri-o://2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833" gracePeriod=600 Apr 24 16:43:38.237107 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.237081 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-857678c95-pmn4r"] Apr 24 16:43:38.239179 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:43:38.239151 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09e332a_bc47_445b_ba0f_8b6740d2419b.slice/crio-403ab3b0bbf56d599d2317759d58a5122bf1a0cc9d39858d3aaab00360acbdb3 WatchSource:0}: Error finding container 403ab3b0bbf56d599d2317759d58a5122bf1a0cc9d39858d3aaab00360acbdb3: Status 404 returned error can't find the container with id 403ab3b0bbf56d599d2317759d58a5122bf1a0cc9d39858d3aaab00360acbdb3 Apr 24 16:43:38.654135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654097 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570" exitCode=0 Apr 24 16:43:38.654135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654124 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1" exitCode=0 Apr 24 16:43:38.654135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654133 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833" exitCode=0 Apr 24 16:43:38.654135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654140 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6" exitCode=0 Apr 24 16:43:38.654135 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654146 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000" exitCode=0 Apr 24 16:43:38.654646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654164 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570"} Apr 24 16:43:38.654646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654195 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1"} Apr 24 16:43:38.654646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654210 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833"} Apr 24 16:43:38.654646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654220 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6"} Apr 24 16:43:38.654646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.654231 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000"} Apr 24 16:43:38.655209 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:38.655180 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" event={"ID":"b09e332a-bc47-445b-ba0f-8b6740d2419b","Type":"ContainerStarted","Data":"403ab3b0bbf56d599d2317759d58a5122bf1a0cc9d39858d3aaab00360acbdb3"} Apr 24 16:43:39.390859 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.390835 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.536321 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536293 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536321 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536322 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536561 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536373 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536561 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536402 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536561 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536425 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536729 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536707 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536793 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536770 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536849 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536814 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:39.536849 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536823 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536954 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536879 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.536954 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536931 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536962 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537141 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537121 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537187 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537155 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537242 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537202 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537242 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537224 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8879j\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537254 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537278 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537312 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle\") pod \"e68f78b9-dc26-43dd-91ba-c06247483ca0\" (UID: \"e68f78b9-dc26-43dd-91ba-c06247483ca0\") " Apr 24 16:43:39.537892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.537587 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.537892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.536928 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:39.539487 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.538297 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:39.539487 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.538443 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:39.539487 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.538982 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out" (OuterVolumeSpecName: "config-out") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:43:39.539487 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.539086 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 24 16:43:39.539772 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.539726 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.539937 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.539905 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:43:39.540086 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.540057 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config" (OuterVolumeSpecName: "config") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.540243 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.540202 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.541446 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.541419 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.541904 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.541879 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:43:39.542250 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.542222 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.542498 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.542437 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.543125 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.543075 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.543291 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.543271 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j" (OuterVolumeSpecName: "kube-api-access-8879j") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "kube-api-access-8879j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:43:39.543980 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.543952 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.554174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.554152 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config" (OuterVolumeSpecName: "web-config") pod "e68f78b9-dc26-43dd-91ba-c06247483ca0" (UID: "e68f78b9-dc26-43dd-91ba-c06247483ca0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:43:39.638433 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638399 2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-metrics-client-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638433 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638430 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-web-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638446 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638459 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638472 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-tls-assets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638483 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8879j\" (UniqueName: \"kubernetes.io/projected/e68f78b9-dc26-43dd-91ba-c06247483ca0-kube-api-access-8879j\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638494 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-db\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638509 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638524 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638536 2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-grpc-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638551 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638566 2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-secret-kube-rbac-proxy\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638577 2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-config\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638588 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e68f78b9-dc26-43dd-91ba-c06247483ca0-config-out\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638599 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638611 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e68f78b9-dc26-43dd-91ba-c06247483ca0-configmap-metrics-client-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.638659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.638624 2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e68f78b9-dc26-43dd-91ba-c06247483ca0-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:43:39.663115 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.663079 2561 generic.go:358] "Generic (PLEG): container finished" podID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerID="f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d" exitCode=0 Apr 24 16:43:39.663547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.663141 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d"} Apr 24 16:43:39.663547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.663176 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e68f78b9-dc26-43dd-91ba-c06247483ca0","Type":"ContainerDied","Data":"4105fdf4a5b31f9b189dde7fed94bceba7f638febf9c4b89ee1453f42226cc27"} Apr 24 16:43:39.663547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.663197 2561 scope.go:117] "RemoveContainer" containerID="5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570" Apr 24 16:43:39.663547 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.663224 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.673202 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.673183 2561 scope.go:117] "RemoveContainer" containerID="0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1" Apr 24 16:43:39.703877 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.703848 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:39.709418 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.709381 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:39.754331 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754297 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:39.754699 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754657 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="prometheus" Apr 24 16:43:39.754699 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754694 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="prometheus" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754706 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="thanos-sidecar" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754715 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="thanos-sidecar" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754724 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="config-reloader" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754730 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="config-reloader" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754741 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="init-config-reloader" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754747 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="init-config-reloader" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754754 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-thanos" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754762 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-thanos" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754774 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754780 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754786 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-web" Apr 24 16:43:39.754856 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754793 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-web" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754862 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-web" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754875 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy-thanos" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754882 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="prometheus" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754887 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="config-reloader" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754892 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="thanos-sidecar" Apr 24 16:43:39.755279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.754900 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" containerName="kube-rbac-proxy" Apr 24 16:43:39.758510 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.758487 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.762064 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.762039 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 24 16:43:39.762581 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.762564 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 24 16:43:39.762581 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.762568 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 24 16:43:39.762872 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.762853 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 24 16:43:39.763156 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.763137 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4lcbz\"" Apr 24 16:43:39.763259 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.763163 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 24 16:43:39.763259 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.763224 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 24 16:43:39.763580 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.763562 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 24 16:43:39.764174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.764140 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 24 16:43:39.767289 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.767271 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-811blobdj7vv8\"" Apr 24 16:43:39.767784 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.767768 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 24 16:43:39.769188 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.769168 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 24 16:43:39.771264 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.771246 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 24 16:43:39.773286 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.773267 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 24 16:43:39.777532 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.777512 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 24 16:43:39.802363 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.802334 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:39.840878 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.840844 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841030 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.840887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrnh\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-kube-api-access-vlrnh\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841030 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.840913 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841030 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.840943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841030 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841007 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841038 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841153 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841219 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841249 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841270 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841392 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.841589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.841435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.851445 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.851414 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68f78b9-dc26-43dd-91ba-c06247483ca0" path="/var/lib/kubelet/pods/e68f78b9-dc26-43dd-91ba-c06247483ca0/volumes" Apr 24 16:43:39.942065 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.941990 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942065 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942026 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942065 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942050 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942358 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942189 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942358 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942288 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942358 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942324 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942358 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942355 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942543 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942391 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrnh\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-kube-api-access-vlrnh\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942887 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942808 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.942887 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942855 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943059 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942920 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943059 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942960 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943059 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.942989 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943059 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943030 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943083 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943108 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943142 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943255 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943166 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.943437 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.943361 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.944773 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.944432 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.945797 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.945767 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946131 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946108 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946229 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946143 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946224 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946441 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946418 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946525 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946505 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946535 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18fd18e7-cf1c-46bf-99b6-518f009ede0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.946836 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.947020 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.946997 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.947195 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.947168 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.947428 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.947333 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.947814 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.947793 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.947888 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.947855 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.948227 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.948211 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18fd18e7-cf1c-46bf-99b6-518f009ede0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.948761 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.948746 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18fd18e7-cf1c-46bf-99b6-518f009ede0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:39.953996 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:39.953975 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrnh\" (UniqueName: \"kubernetes.io/projected/18fd18e7-cf1c-46bf-99b6-518f009ede0e-kube-api-access-vlrnh\") pod \"prometheus-k8s-0\" (UID: \"18fd18e7-cf1c-46bf-99b6-518f009ede0e\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:40.070126 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.070087 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:43:40.107131 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.107104 2561 scope.go:117] "RemoveContainer" containerID="f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d" Apr 24 16:43:40.125987 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.125964 2561 scope.go:117] "RemoveContainer" containerID="2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833" Apr 24 16:43:40.160546 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.160518 2561 scope.go:117] "RemoveContainer" containerID="e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6" Apr 24 16:43:40.169401 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.169383 2561 scope.go:117] "RemoveContainer" containerID="ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000" Apr 24 16:43:40.176799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.176779 2561 scope.go:117] "RemoveContainer" containerID="b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667" Apr 24 16:43:40.183792 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.183768 2561 scope.go:117] "RemoveContainer" containerID="5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570" Apr 24 16:43:40.184109 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.184086 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570\": container with ID starting with 5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570 not found: ID does not exist" containerID="5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570" Apr 24 16:43:40.184207 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184121 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570"} err="failed to get container status \"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570\": rpc error: code = NotFound desc = could not find container \"5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570\": container with ID starting with 5f8154c121f4c0f411dc4befe4615652fa5062d3d27fd89623668243b2b13570 not found: ID does not exist" Apr 24 16:43:40.184207 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184146 2561 scope.go:117] "RemoveContainer" containerID="0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1" Apr 24 16:43:40.184563 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.184514 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1\": container with ID starting with 0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1 not found: ID does not exist" containerID="0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1" Apr 24 16:43:40.184563 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184551 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1"} err="failed to get container status \"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1\": rpc error: code = NotFound desc = could not find container \"0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1\": container with ID starting with 0e6e13fe2be7679c810bbc28e863a61d2547f87d57cd7a91d931a38127e305c1 not found: ID does not exist" Apr 24 16:43:40.184785 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184576 2561 scope.go:117] "RemoveContainer" containerID="f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d" Apr 24 16:43:40.184911 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.184879 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d\": container with ID starting with f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d not found: ID does not exist" containerID="f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d" Apr 24 16:43:40.184973 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184912 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d"} err="failed to get container status \"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d\": rpc error: code = NotFound desc = could not find container \"f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d\": container with ID starting with f6f7bb5e7d63e0e8cc666f94b3dd20f42b30e35321028b085cb145a126db0b5d not found: ID does not exist" Apr 24 16:43:40.184973 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.184936 2561 scope.go:117] "RemoveContainer" containerID="2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833" Apr 24 16:43:40.185193 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.185172 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833\": container with ID starting with 2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833 not found: ID does not exist" containerID="2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833" Apr 24 16:43:40.185281 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185200 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833"} err="failed to get container status \"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833\": rpc error: code = NotFound desc = could not find container \"2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833\": container with ID starting with 2f23ccbb66fddd2e70a5de8cc57f48e3dde0fdc11f1aacf17a6fd4f804905833 not found: ID does not exist" Apr 24 16:43:40.185281 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185220 2561 scope.go:117] "RemoveContainer" containerID="e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6" Apr 24 16:43:40.185503 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.185480 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6\": container with ID starting with e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6 not found: ID does not exist" containerID="e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6" Apr 24 16:43:40.185559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185511 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6"} err="failed to get container status \"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6\": rpc error: code = NotFound desc = could not find container \"e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6\": container with ID starting with e5b02c6e562c9a4087597ad718ea3fdc616f6b547e47269aa56412e27b1f52a6 not found: ID does not exist" Apr 24 16:43:40.185559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185533 2561 scope.go:117] "RemoveContainer" containerID="ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000" Apr 24 16:43:40.185873 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.185808 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000\": container with ID starting with ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000 not found: ID does not exist" containerID="ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000" Apr 24 16:43:40.185873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185834 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000"} err="failed to get container status \"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000\": rpc error: code = NotFound desc = could not find container \"ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000\": container with ID starting with ce4807267f2acb92b70c3937350806494a83141315ff2d25f4cd8743b6562000 not found: ID does not exist" Apr 24 16:43:40.185873 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.185854 2561 scope.go:117] "RemoveContainer" containerID="b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667" Apr 24 16:43:40.186142 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:43:40.186118 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667\": container with ID starting with b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667 not found: ID does not exist" containerID="b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667" Apr 24 16:43:40.186203 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.186149 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667"} err="failed to get container status \"b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667\": rpc error: code = NotFound desc = could not find container \"b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667\": container with ID starting with b16c3ec8d3b51e32a85a917e576054218d3f8e277ffb091032244b9fbf572667 not found: ID does not exist" Apr 24 16:43:40.251158 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.251133 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 24 16:43:40.253330 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:43:40.253302 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fd18e7_cf1c_46bf_99b6_518f009ede0e.slice/crio-0745a49680abc8c7c8a14b91ead44fd5406fff9575422fad4996605982c28e27 WatchSource:0}: Error finding container 0745a49680abc8c7c8a14b91ead44fd5406fff9575422fad4996605982c28e27: Status 404 returned error can't find the container with id 0745a49680abc8c7c8a14b91ead44fd5406fff9575422fad4996605982c28e27 Apr 24 16:43:40.667934 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.667899 2561 generic.go:358] "Generic (PLEG): container finished" podID="18fd18e7-cf1c-46bf-99b6-518f009ede0e" containerID="a75332e4628eedb075c45fe94a6be180b6c17a758c8b2b6f0a54b60ba3ab8ebf" exitCode=0 Apr 24 16:43:40.668334 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.667985 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerDied","Data":"a75332e4628eedb075c45fe94a6be180b6c17a758c8b2b6f0a54b60ba3ab8ebf"} Apr 24 16:43:40.668334 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.668020 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"0745a49680abc8c7c8a14b91ead44fd5406fff9575422fad4996605982c28e27"} Apr 24 16:43:40.670646 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.670625 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" event={"ID":"b09e332a-bc47-445b-ba0f-8b6740d2419b","Type":"ContainerStarted","Data":"b16d83b5f93fd2dd8e23a01a8e3f1ba384fb1f01aeb4db405545c22e69f909b1"} Apr 24 16:43:40.670769 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.670651 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" event={"ID":"b09e332a-bc47-445b-ba0f-8b6740d2419b","Type":"ContainerStarted","Data":"8c2f66e4b1c90360f1509c5e88b86f39433e1ebb551e14eaf0cbc9e2bace2fbe"} Apr 24 16:43:40.670769 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.670678 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" event={"ID":"b09e332a-bc47-445b-ba0f-8b6740d2419b","Type":"ContainerStarted","Data":"dd9a52a55cb2e6479a399c30c845f3c8de3b050c30fc1ba66585f8fccd93aaca"} Apr 24 16:43:40.745176 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:40.745125 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-857678c95-pmn4r" podStartSLOduration=1.836231328 podStartE2EDuration="3.745107226s" podCreationTimestamp="2026-04-24 16:43:37 +0000 UTC" firstStartedPulling="2026-04-24 16:43:38.264879681 +0000 UTC m=+259.014050434" lastFinishedPulling="2026-04-24 16:43:40.173755591 +0000 UTC m=+260.922926332" observedRunningTime="2026-04-24 16:43:40.743899788 +0000 UTC m=+261.493070549" watchObservedRunningTime="2026-04-24 16:43:40.745107226 +0000 UTC m=+261.494277989" Apr 24 16:43:41.676076 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676039 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"8b03a18589b93ab0b0e20fece2e6a6e980f6df1bf17c0b543549b59010fcbeaf"} Apr 24 16:43:41.676076 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676081 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"8938524051d21cd90ca04dc49aab6703bc3aec6e5431bc24e0b1a21b74da615c"} Apr 24 16:43:41.676467 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676091 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"fb1b0f2abc63032a90d60e2b46dfd6d9814cfe608b6d95001084c548d87a2525"} Apr 24 16:43:41.676467 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676100 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"89890ddd64a464c938b93fd37d657a33804e4b1ed15da879a8a38d0058bf478e"} Apr 24 16:43:41.676467 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676108 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"638e9156c6bcd265cd216eb1b46af5d44bd0a61a4a5b0fc930f7a9feda902548"} Apr 24 16:43:41.676467 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.676117 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18fd18e7-cf1c-46bf-99b6-518f009ede0e","Type":"ContainerStarted","Data":"862b6faf82d2ea9d52a95b9f03c902ea3d30dccd7a465dcd338a6608c7899566"} Apr 24 16:43:41.727740 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:41.727692 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.727677097 podStartE2EDuration="2.727677097s" podCreationTimestamp="2026-04-24 16:43:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:43:41.727215897 +0000 UTC m=+262.476386669" watchObservedRunningTime="2026-04-24 16:43:41.727677097 +0000 UTC m=+262.476847850" Apr 24 16:43:45.071055 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:43:45.071022 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:44:19.770044 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:19.770018 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:44:19.771169 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:19.771142 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:44:19.776087 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:19.776072 2561 kubelet.go:1628] "Image garbage collection succeeded" Apr 24 16:44:40.071023 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:40.070990 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:44:40.086073 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:40.086045 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:44:40.859466 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:44:40.859435 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 24 16:47:27.817328 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.817291 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-ljhhd"] Apr 24 16:47:27.820762 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.820741 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.823096 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.823077 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Apr 24 16:47:27.823179 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.823145 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jlbhm\"" Apr 24 16:47:27.823970 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.823956 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Apr 24 16:47:27.832528 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.832505 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-ljhhd"] Apr 24 16:47:27.839447 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.839426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnphh\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-kube-api-access-hnphh\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.839533 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.839469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.940403 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.940369 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnphh\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-kube-api-access-hnphh\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.940403 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.940409 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.949764 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.949737 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:27.949940 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:27.949922 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnphh\" (UniqueName: \"kubernetes.io/projected/e8f50367-1f51-41e7-a140-83513332e077-kube-api-access-hnphh\") pod \"cert-manager-webhook-587ccfb98-ljhhd\" (UID: \"e8f50367-1f51-41e7-a140-83513332e077\") " pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:28.140127 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:28.140100 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:28.257264 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:28.257229 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-ljhhd"] Apr 24 16:47:28.261089 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:47:28.261062 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f50367_1f51_41e7_a140_83513332e077.slice/crio-11f92b35fe5fb3ceabddaddc9dbe1642cf5caba102ddfe132e0bf7b7e48eee21 WatchSource:0}: Error finding container 11f92b35fe5fb3ceabddaddc9dbe1642cf5caba102ddfe132e0bf7b7e48eee21: Status 404 returned error can't find the container with id 11f92b35fe5fb3ceabddaddc9dbe1642cf5caba102ddfe132e0bf7b7e48eee21 Apr 24 16:47:28.262886 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:28.262870 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 16:47:28.323851 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:28.323816 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" event={"ID":"e8f50367-1f51-41e7-a140-83513332e077","Type":"ContainerStarted","Data":"11f92b35fe5fb3ceabddaddc9dbe1642cf5caba102ddfe132e0bf7b7e48eee21"} Apr 24 16:47:31.335996 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:31.335960 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" event={"ID":"e8f50367-1f51-41e7-a140-83513332e077","Type":"ContainerStarted","Data":"70a837dec07f7be912a4e468873e26f290773693f6f4d65cee29f713c735f4cf"} Apr 24 16:47:31.336335 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:31.336086 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:47:31.356681 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:31.356626 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" podStartSLOduration=1.501056906 podStartE2EDuration="4.356615418s" podCreationTimestamp="2026-04-24 16:47:27 +0000 UTC" firstStartedPulling="2026-04-24 16:47:28.262996576 +0000 UTC m=+489.012167315" lastFinishedPulling="2026-04-24 16:47:31.118555085 +0000 UTC m=+491.867725827" observedRunningTime="2026-04-24 16:47:31.354809332 +0000 UTC m=+492.103980092" watchObservedRunningTime="2026-04-24 16:47:31.356615418 +0000 UTC m=+492.105786177" Apr 24 16:47:37.340663 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:47:37.340628 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-ljhhd" Apr 24 16:48:10.006225 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.006145 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm"] Apr 24 16:48:10.009016 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.008995 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.011660 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.011637 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6xgxr\"" Apr 24 16:48:10.012644 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.012625 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Apr 24 16:48:10.013013 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.012992 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Apr 24 16:48:10.013234 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.013217 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Apr 24 16:48:10.013453 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.013369 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Apr 24 16:48:10.013767 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.013754 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Apr 24 16:48:10.032343 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.032323 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm"] Apr 24 16:48:10.085123 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.085098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95eb5291-941d-4566-bba9-e5bbbcd96c88-manager-config\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.085237 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.085127 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x9n\" (UniqueName: \"kubernetes.io/projected/95eb5291-941d-4566-bba9-e5bbbcd96c88-kube-api-access-f7x9n\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.085237 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.085167 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.085237 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.085224 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-metrics-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.185895 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.185851 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-metrics-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.186062 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.185920 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95eb5291-941d-4566-bba9-e5bbbcd96c88-manager-config\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.186102 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.186075 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x9n\" (UniqueName: \"kubernetes.io/projected/95eb5291-941d-4566-bba9-e5bbbcd96c88-kube-api-access-f7x9n\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.186143 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.186130 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.186512 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.186493 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95eb5291-941d-4566-bba9-e5bbbcd96c88-manager-config\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.188339 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.188320 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.188448 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.188428 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95eb5291-941d-4566-bba9-e5bbbcd96c88-metrics-cert\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.214426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.214401 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x9n\" (UniqueName: \"kubernetes.io/projected/95eb5291-941d-4566-bba9-e5bbbcd96c88-kube-api-access-f7x9n\") pod \"lws-controller-manager-75b5bf9f6d-t87zm\" (UID: \"95eb5291-941d-4566-bba9-e5bbbcd96c88\") " pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.321473 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.321402 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:10.444384 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:10.444358 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm"] Apr 24 16:48:10.447051 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:48:10.447027 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eb5291_941d_4566_bba9_e5bbbcd96c88.slice/crio-cb96ff4c47ddde279f9873abccf998657e5de811cff97e547d381a48f1f5d463 WatchSource:0}: Error finding container cb96ff4c47ddde279f9873abccf998657e5de811cff97e547d381a48f1f5d463: Status 404 returned error can't find the container with id cb96ff4c47ddde279f9873abccf998657e5de811cff97e547d381a48f1f5d463 Apr 24 16:48:11.451775 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:11.451722 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" event={"ID":"95eb5291-941d-4566-bba9-e5bbbcd96c88","Type":"ContainerStarted","Data":"cb96ff4c47ddde279f9873abccf998657e5de811cff97e547d381a48f1f5d463"} Apr 24 16:48:13.459515 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:13.459476 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" event={"ID":"95eb5291-941d-4566-bba9-e5bbbcd96c88","Type":"ContainerStarted","Data":"7d24ecd8f7753ba7db5ac408737bcff7102507fe1389ca7a8d298bea7afce173"} Apr 24 16:48:13.459921 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:13.459597 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:13.485302 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:13.485226 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" podStartSLOduration=1.965928809 podStartE2EDuration="4.485213643s" podCreationTimestamp="2026-04-24 16:48:09 +0000 UTC" firstStartedPulling="2026-04-24 16:48:10.448718759 +0000 UTC m=+531.197889498" lastFinishedPulling="2026-04-24 16:48:12.968003589 +0000 UTC m=+533.717174332" observedRunningTime="2026-04-24 16:48:13.484773628 +0000 UTC m=+534.233944416" watchObservedRunningTime="2026-04-24 16:48:13.485213643 +0000 UTC m=+534.234384403" Apr 24 16:48:21.688506 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.688471 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4"] Apr 24 16:48:21.691882 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.691866 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.694577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.694552 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-clh59\"" Apr 24 16:48:21.694780 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.694759 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"istio-ca-root-cert\"" Apr 24 16:48:21.704899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.704876 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4"] Apr 24 16:48:21.781648 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781620 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781819 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781677 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781819 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781819 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781819 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781829 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c144f2dd-e507-4c01-98b6-489818cb43f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.781985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.781924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9v49\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-kube-api-access-j9v49\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883079 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883050 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883090 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883116 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883145 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883177 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883206 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883228 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883265 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c144f2dd-e507-4c01-98b6-489818cb43f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883426 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883292 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9v49\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-kube-api-access-j9v49\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883578 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883446 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883578 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883561 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883706 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883630 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.883906 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883883 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.884017 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.883958 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c144f2dd-e507-4c01-98b6-489818cb43f9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.885419 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.885391 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.885604 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.885586 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.891820 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.891797 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:21.892014 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:21.891998 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9v49\" (UniqueName: \"kubernetes.io/projected/c144f2dd-e507-4c01-98b6-489818cb43f9-kube-api-access-j9v49\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-5lfr4\" (UID: \"c144f2dd-e507-4c01-98b6-489818cb43f9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:22.004587 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:22.004515 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:22.125719 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:22.125696 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4"] Apr 24 16:48:22.127436 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:48:22.127406 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc144f2dd_e507_4c01_98b6_489818cb43f9.slice/crio-7ce7fad61fd246dd355294f331a6fe607e48e279ec920e4e135d398e56d06ec7 WatchSource:0}: Error finding container 7ce7fad61fd246dd355294f331a6fe607e48e279ec920e4e135d398e56d06ec7: Status 404 returned error can't find the container with id 7ce7fad61fd246dd355294f331a6fe607e48e279ec920e4e135d398e56d06ec7 Apr 24 16:48:22.487413 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:22.487381 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" event={"ID":"c144f2dd-e507-4c01-98b6-489818cb43f9","Type":"ContainerStarted","Data":"7ce7fad61fd246dd355294f331a6fe607e48e279ec920e4e135d398e56d06ec7"} Apr 24 16:48:24.318884 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.318851 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:48:24.319138 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.318920 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:48:24.319138 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.318945 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:48:24.464423 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.464387 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75b5bf9f6d-t87zm" Apr 24 16:48:24.497170 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.497133 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" event={"ID":"c144f2dd-e507-4c01-98b6-489818cb43f9","Type":"ContainerStarted","Data":"e364821659e1e8a1d85d95332d432bfa071eb38c828ef93c420f02825a6dd594"} Apr 24 16:48:24.519102 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:24.519044 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" podStartSLOduration=1.32994488 podStartE2EDuration="3.51902568s" podCreationTimestamp="2026-04-24 16:48:21 +0000 UTC" firstStartedPulling="2026-04-24 16:48:22.129516781 +0000 UTC m=+542.878687522" lastFinishedPulling="2026-04-24 16:48:24.318597579 +0000 UTC m=+545.067768322" observedRunningTime="2026-04-24 16:48:24.517837285 +0000 UTC m=+545.267008085" watchObservedRunningTime="2026-04-24 16:48:24.51902568 +0000 UTC m=+545.268196439" Apr 24 16:48:25.005360 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:25.005328 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:25.009999 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:25.009969 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:25.500566 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:25.500536 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:25.501473 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:25.501455 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-5lfr4" Apr 24 16:48:47.616546 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.616507 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r"] Apr 24 16:48:47.627685 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.627650 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:47.630361 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.630333 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Apr 24 16:48:47.630506 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.630483 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-smpg8\"" Apr 24 16:48:47.630575 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.630493 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Apr 24 16:48:47.631491 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.631470 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Apr 24 16:48:47.633128 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.633106 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r"] Apr 24 16:48:47.703023 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.702992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xth\" (UniqueName: \"kubernetes.io/projected/0db6e30d-4e12-4301-af19-38fb78346d82-kube-api-access-m9xth\") pod \"dns-operator-controller-manager-844548ff4c-9ht4r\" (UID: \"0db6e30d-4e12-4301-af19-38fb78346d82\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:47.803384 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.803353 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xth\" (UniqueName: \"kubernetes.io/projected/0db6e30d-4e12-4301-af19-38fb78346d82-kube-api-access-m9xth\") pod \"dns-operator-controller-manager-844548ff4c-9ht4r\" (UID: \"0db6e30d-4e12-4301-af19-38fb78346d82\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:47.815031 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.815006 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xth\" (UniqueName: \"kubernetes.io/projected/0db6e30d-4e12-4301-af19-38fb78346d82-kube-api-access-m9xth\") pod \"dns-operator-controller-manager-844548ff4c-9ht4r\" (UID: \"0db6e30d-4e12-4301-af19-38fb78346d82\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:47.937829 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:47.937745 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:48.073708 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:48.073657 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r"] Apr 24 16:48:48.077969 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:48:48.077929 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db6e30d_4e12_4301_af19_38fb78346d82.slice/crio-423c8bac55a2a2207ee55224f091e35933818a96fe40b17fee8e2b5f9115334b WatchSource:0}: Error finding container 423c8bac55a2a2207ee55224f091e35933818a96fe40b17fee8e2b5f9115334b: Status 404 returned error can't find the container with id 423c8bac55a2a2207ee55224f091e35933818a96fe40b17fee8e2b5f9115334b Apr 24 16:48:48.567949 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:48.567909 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" event={"ID":"0db6e30d-4e12-4301-af19-38fb78346d82","Type":"ContainerStarted","Data":"423c8bac55a2a2207ee55224f091e35933818a96fe40b17fee8e2b5f9115334b"} Apr 24 16:48:49.689742 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.689706 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d"] Apr 24 16:48:49.693169 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.693144 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:49.695650 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.695628 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5p2kh\"" Apr 24 16:48:49.706370 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.706343 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d"] Apr 24 16:48:49.720161 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.720129 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbxj\" (UniqueName: \"kubernetes.io/projected/c14c9346-0f21-4225-a096-22fe8163a745-kube-api-access-pjbxj\") pod \"limitador-operator-controller-manager-c7fb4c8d5-ftb2d\" (UID: \"c14c9346-0f21-4225-a096-22fe8163a745\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:49.821284 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.821248 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbxj\" (UniqueName: \"kubernetes.io/projected/c14c9346-0f21-4225-a096-22fe8163a745-kube-api-access-pjbxj\") pod \"limitador-operator-controller-manager-c7fb4c8d5-ftb2d\" (UID: \"c14c9346-0f21-4225-a096-22fe8163a745\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:49.832283 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:49.832254 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbxj\" (UniqueName: \"kubernetes.io/projected/c14c9346-0f21-4225-a096-22fe8163a745-kube-api-access-pjbxj\") pod \"limitador-operator-controller-manager-c7fb4c8d5-ftb2d\" (UID: \"c14c9346-0f21-4225-a096-22fe8163a745\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:50.006588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:50.006504 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:51.123334 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:51.123313 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d"] Apr 24 16:48:51.125352 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:48:51.125321 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14c9346_0f21_4225_a096_22fe8163a745.slice/crio-f8443b44108629ae7915d01ed149e81f2c9fa1ca610d10bdb7de68ece54f7e3b WatchSource:0}: Error finding container f8443b44108629ae7915d01ed149e81f2c9fa1ca610d10bdb7de68ece54f7e3b: Status 404 returned error can't find the container with id f8443b44108629ae7915d01ed149e81f2c9fa1ca610d10bdb7de68ece54f7e3b Apr 24 16:48:51.581601 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:51.581567 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" event={"ID":"0db6e30d-4e12-4301-af19-38fb78346d82","Type":"ContainerStarted","Data":"a1dca8635746687d42b42876bbac709d2e6262ed2dbb0cab7eb374542aeef3a3"} Apr 24 16:48:51.581790 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:51.581630 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:48:51.582684 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:51.582647 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" event={"ID":"c14c9346-0f21-4225-a096-22fe8163a745","Type":"ContainerStarted","Data":"f8443b44108629ae7915d01ed149e81f2c9fa1ca610d10bdb7de68ece54f7e3b"} Apr 24 16:48:53.594252 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:53.594208 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" event={"ID":"c14c9346-0f21-4225-a096-22fe8163a745","Type":"ContainerStarted","Data":"2309ce140329c8a9ccd4eb95a5efbb876aa66fbbdb59555dfcf4d7c5d0ce78c3"} Apr 24 16:48:53.594724 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:53.594356 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:48:53.619330 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:53.619282 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" podStartSLOduration=3.643189283 podStartE2EDuration="6.619269244s" podCreationTimestamp="2026-04-24 16:48:47 +0000 UTC" firstStartedPulling="2026-04-24 16:48:48.082966629 +0000 UTC m=+568.832137368" lastFinishedPulling="2026-04-24 16:48:51.05904659 +0000 UTC m=+571.808217329" observedRunningTime="2026-04-24 16:48:51.638115975 +0000 UTC m=+572.387286736" watchObservedRunningTime="2026-04-24 16:48:53.619269244 +0000 UTC m=+574.368440006" Apr 24 16:48:53.619490 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:48:53.619365 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" podStartSLOduration=2.84949914 podStartE2EDuration="4.61936119s" podCreationTimestamp="2026-04-24 16:48:49 +0000 UTC" firstStartedPulling="2026-04-24 16:48:51.127915215 +0000 UTC m=+571.877085954" lastFinishedPulling="2026-04-24 16:48:52.897777254 +0000 UTC m=+573.646948004" observedRunningTime="2026-04-24 16:48:53.617938661 +0000 UTC m=+574.367109421" watchObservedRunningTime="2026-04-24 16:48:53.61936119 +0000 UTC m=+574.368531951" Apr 24 16:49:02.591591 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:02.591560 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-9ht4r" Apr 24 16:49:04.600869 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:04.600837 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-ftb2d" Apr 24 16:49:19.796692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:19.796652 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:49:19.797115 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:19.796758 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:49:35.758187 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.758113 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:35.761385 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.761366 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:35.763563 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.763543 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s5znh\"" Apr 24 16:49:35.768992 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.768969 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:35.810397 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.810365 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgrp\" (UniqueName: \"kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp\") pod \"authorino-674b59b84c-2pdb8\" (UID: \"da3fc203-2fcc-4ac1-95b3-bee652106094\") " pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:35.911639 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.911601 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgrp\" (UniqueName: \"kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp\") pod \"authorino-674b59b84c-2pdb8\" (UID: \"da3fc203-2fcc-4ac1-95b3-bee652106094\") " pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:35.928955 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:35.928925 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgrp\" (UniqueName: \"kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp\") pod \"authorino-674b59b84c-2pdb8\" (UID: \"da3fc203-2fcc-4ac1-95b3-bee652106094\") " pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:36.071992 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:36.071914 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:36.188092 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:36.188063 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:36.190906 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:49:36.190884 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda3fc203_2fcc_4ac1_95b3_bee652106094.slice/crio-935a20a4aa420e3d4ead483b6a1f9fd744605b33ff1060835d37ff48c90f9f53 WatchSource:0}: Error finding container 935a20a4aa420e3d4ead483b6a1f9fd744605b33ff1060835d37ff48c90f9f53: Status 404 returned error can't find the container with id 935a20a4aa420e3d4ead483b6a1f9fd744605b33ff1060835d37ff48c90f9f53 Apr 24 16:49:36.735833 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:36.735792 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-2pdb8" event={"ID":"da3fc203-2fcc-4ac1-95b3-bee652106094","Type":"ContainerStarted","Data":"935a20a4aa420e3d4ead483b6a1f9fd744605b33ff1060835d37ff48c90f9f53"} Apr 24 16:49:39.747480 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:39.747441 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-2pdb8" event={"ID":"da3fc203-2fcc-4ac1-95b3-bee652106094","Type":"ContainerStarted","Data":"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c"} Apr 24 16:49:39.763679 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:39.763631 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-2pdb8" podStartSLOduration=1.739363219 podStartE2EDuration="4.763613345s" podCreationTimestamp="2026-04-24 16:49:35 +0000 UTC" firstStartedPulling="2026-04-24 16:49:36.192080402 +0000 UTC m=+616.941251145" lastFinishedPulling="2026-04-24 16:49:39.216330532 +0000 UTC m=+619.965501271" observedRunningTime="2026-04-24 16:49:39.762807977 +0000 UTC m=+620.511978749" watchObservedRunningTime="2026-04-24 16:49:39.763613345 +0000 UTC m=+620.512784106" Apr 24 16:49:39.937036 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:39.937007 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:41.754655 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:41.754611 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-2pdb8" podUID="da3fc203-2fcc-4ac1-95b3-bee652106094" containerName="authorino" containerID="cri-o://27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c" gracePeriod=30 Apr 24 16:49:41.989898 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:41.989875 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:42.060456 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.060374 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsgrp\" (UniqueName: \"kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp\") pod \"da3fc203-2fcc-4ac1-95b3-bee652106094\" (UID: \"da3fc203-2fcc-4ac1-95b3-bee652106094\") " Apr 24 16:49:42.062455 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.062427 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp" (OuterVolumeSpecName: "kube-api-access-zsgrp") pod "da3fc203-2fcc-4ac1-95b3-bee652106094" (UID: "da3fc203-2fcc-4ac1-95b3-bee652106094"). InnerVolumeSpecName "kube-api-access-zsgrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:49:42.161303 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.161276 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgrp\" (UniqueName: \"kubernetes.io/projected/da3fc203-2fcc-4ac1-95b3-bee652106094-kube-api-access-zsgrp\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:49:42.759149 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.759109 2561 generic.go:358] "Generic (PLEG): container finished" podID="da3fc203-2fcc-4ac1-95b3-bee652106094" containerID="27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c" exitCode=0 Apr 24 16:49:42.759588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.759165 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-2pdb8" Apr 24 16:49:42.759588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.759196 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-2pdb8" event={"ID":"da3fc203-2fcc-4ac1-95b3-bee652106094","Type":"ContainerDied","Data":"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c"} Apr 24 16:49:42.759588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.759237 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-2pdb8" event={"ID":"da3fc203-2fcc-4ac1-95b3-bee652106094","Type":"ContainerDied","Data":"935a20a4aa420e3d4ead483b6a1f9fd744605b33ff1060835d37ff48c90f9f53"} Apr 24 16:49:42.759588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.759252 2561 scope.go:117] "RemoveContainer" containerID="27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c" Apr 24 16:49:42.770551 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.770528 2561 scope.go:117] "RemoveContainer" containerID="27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c" Apr 24 16:49:42.771185 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:49:42.770938 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c\": container with ID starting with 27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c not found: ID does not exist" containerID="27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c" Apr 24 16:49:42.771185 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.770979 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c"} err="failed to get container status \"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c\": rpc error: code = NotFound desc = could not find container \"27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c\": container with ID starting with 27ca0b92a65a3fc156c7dd181fdd7bf3b0f7b784370f6dc86d70bd90634ef30c not found: ID does not exist" Apr 24 16:49:42.792280 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.792254 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:42.795748 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:42.795726 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-2pdb8"] Apr 24 16:49:43.850244 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:43.850204 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3fc203-2fcc-4ac1-95b3-bee652106094" path="/var/lib/kubelet/pods/da3fc203-2fcc-4ac1-95b3-bee652106094/volumes" Apr 24 16:49:58.348164 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.348130 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-lv7qb"] Apr 24 16:49:58.348574 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.348471 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3fc203-2fcc-4ac1-95b3-bee652106094" containerName="authorino" Apr 24 16:49:58.348574 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.348482 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3fc203-2fcc-4ac1-95b3-bee652106094" containerName="authorino" Apr 24 16:49:58.348574 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.348553 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="da3fc203-2fcc-4ac1-95b3-bee652106094" containerName="authorino" Apr 24 16:49:58.352872 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.352855 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.355893 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.355860 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s5znh\"" Apr 24 16:49:58.356039 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.355995 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Apr 24 16:49:58.358340 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.358313 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-lv7qb"] Apr 24 16:49:58.417778 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.417746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66b93258-4aa0-400c-b3c5-01332849494b-tls-cert\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.417892 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.417859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6l8\" (UniqueName: \"kubernetes.io/projected/66b93258-4aa0-400c-b3c5-01332849494b-kube-api-access-dk6l8\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.518238 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.518209 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6l8\" (UniqueName: \"kubernetes.io/projected/66b93258-4aa0-400c-b3c5-01332849494b-kube-api-access-dk6l8\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.518377 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.518262 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66b93258-4aa0-400c-b3c5-01332849494b-tls-cert\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.520504 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.520482 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66b93258-4aa0-400c-b3c5-01332849494b-tls-cert\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.526618 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.526597 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6l8\" (UniqueName: \"kubernetes.io/projected/66b93258-4aa0-400c-b3c5-01332849494b-kube-api-access-dk6l8\") pod \"authorino-68bd676465-lv7qb\" (UID: \"66b93258-4aa0-400c-b3c5-01332849494b\") " pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.662744 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.662704 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-lv7qb" Apr 24 16:49:58.782548 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.782518 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-lv7qb"] Apr 24 16:49:58.785677 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:49:58.785637 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66b93258_4aa0_400c_b3c5_01332849494b.slice/crio-99c7fb09a44eade8b6fea061ac72c71f97161a3505a54d7c9a9e8974b850b082 WatchSource:0}: Error finding container 99c7fb09a44eade8b6fea061ac72c71f97161a3505a54d7c9a9e8974b850b082: Status 404 returned error can't find the container with id 99c7fb09a44eade8b6fea061ac72c71f97161a3505a54d7c9a9e8974b850b082 Apr 24 16:49:58.817559 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:58.817523 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-lv7qb" event={"ID":"66b93258-4aa0-400c-b3c5-01332849494b","Type":"ContainerStarted","Data":"99c7fb09a44eade8b6fea061ac72c71f97161a3505a54d7c9a9e8974b850b082"} Apr 24 16:49:59.821752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:59.821720 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-lv7qb" event={"ID":"66b93258-4aa0-400c-b3c5-01332849494b","Type":"ContainerStarted","Data":"9da96a8586d54046844d337f1b41f81649ea4afca9aae33e841d6948fab1da62"} Apr 24 16:49:59.839047 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:49:59.839005 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-lv7qb" podStartSLOduration=1.440759096 podStartE2EDuration="1.838991876s" podCreationTimestamp="2026-04-24 16:49:58 +0000 UTC" firstStartedPulling="2026-04-24 16:49:58.786931312 +0000 UTC m=+639.536102051" lastFinishedPulling="2026-04-24 16:49:59.185164087 +0000 UTC m=+639.934334831" observedRunningTime="2026-04-24 16:49:59.837388539 +0000 UTC m=+640.586559301" watchObservedRunningTime="2026-04-24 16:49:59.838991876 +0000 UTC m=+640.588162637" Apr 24 16:51:31.888743 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.888658 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-h5kmf"] Apr 24 16:51:31.892091 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.892074 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:31.895009 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.894983 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Apr 24 16:51:31.895121 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.895031 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 24 16:51:31.895336 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.895317 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 24 16:51:31.896210 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.896190 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-f6vhb\"" Apr 24 16:51:31.902651 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.902631 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-h5kmf"] Apr 24 16:51:31.993644 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.993612 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854369b7-40d4-4e3a-9def-ccd9c921065e-cert\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:31.993822 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:31.993652 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7v5\" (UniqueName: \"kubernetes.io/projected/854369b7-40d4-4e3a-9def-ccd9c921065e-kube-api-access-bm7v5\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.094799 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.094771 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854369b7-40d4-4e3a-9def-ccd9c921065e-cert\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.094962 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.094808 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7v5\" (UniqueName: \"kubernetes.io/projected/854369b7-40d4-4e3a-9def-ccd9c921065e-kube-api-access-bm7v5\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.097069 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.097049 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/854369b7-40d4-4e3a-9def-ccd9c921065e-cert\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.110814 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.110788 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7v5\" (UniqueName: \"kubernetes.io/projected/854369b7-40d4-4e3a-9def-ccd9c921065e-kube-api-access-bm7v5\") pod \"odh-model-controller-696fc77849-h5kmf\" (UID: \"854369b7-40d4-4e3a-9def-ccd9c921065e\") " pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.202347 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.202254 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:32.322589 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:32.322564 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-h5kmf"] Apr 24 16:51:32.324710 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:51:32.324654 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854369b7_40d4_4e3a_9def_ccd9c921065e.slice/crio-9b880847409512deb306fe6fccb3fa333a942aea4baeb78259ca775bf29586e7 WatchSource:0}: Error finding container 9b880847409512deb306fe6fccb3fa333a942aea4baeb78259ca775bf29586e7: Status 404 returned error can't find the container with id 9b880847409512deb306fe6fccb3fa333a942aea4baeb78259ca775bf29586e7 Apr 24 16:51:33.122776 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:33.122732 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-h5kmf" event={"ID":"854369b7-40d4-4e3a-9def-ccd9c921065e","Type":"ContainerStarted","Data":"9b880847409512deb306fe6fccb3fa333a942aea4baeb78259ca775bf29586e7"} Apr 24 16:51:35.132257 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:35.132222 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-h5kmf" event={"ID":"854369b7-40d4-4e3a-9def-ccd9c921065e","Type":"ContainerStarted","Data":"c68dba33cb75c2f23caab6c2c46efa1da045649e781a387de4203f7e8bac2dd8"} Apr 24 16:51:35.132629 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:35.132369 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:51:35.149792 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:35.149747 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-h5kmf" podStartSLOduration=1.661413795 podStartE2EDuration="4.149732813s" podCreationTimestamp="2026-04-24 16:51:31 +0000 UTC" firstStartedPulling="2026-04-24 16:51:32.326306296 +0000 UTC m=+733.075477036" lastFinishedPulling="2026-04-24 16:51:34.814625297 +0000 UTC m=+735.563796054" observedRunningTime="2026-04-24 16:51:35.148928947 +0000 UTC m=+735.898099709" watchObservedRunningTime="2026-04-24 16:51:35.149732813 +0000 UTC m=+735.898903573" Apr 24 16:51:46.139808 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:51:46.139778 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-h5kmf" Apr 24 16:52:07.408661 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.408626 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql"] Apr 24 16:52:07.412289 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.412271 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.415281 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.415256 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-s84bt\"" Apr 24 16:52:07.415413 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.415263 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"istio-ca-root-cert\"" Apr 24 16:52:07.415413 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.415309 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Apr 24 16:52:07.415413 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.415374 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Apr 24 16:52:07.426419 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.426395 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql"] Apr 24 16:52:07.497175 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43233bc0-0078-446d-b670-b8c8159a4a95-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497257 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497290 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43233bc0-0078-446d-b670-b8c8159a4a95-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497355 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497582 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497357 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497582 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.497582 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.497464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr727\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-kube-api-access-rr727\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598335 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598250 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598504 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598364 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rr727\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-kube-api-access-rr727\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598504 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598392 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598504 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598478 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43233bc0-0078-446d-b670-b8c8159a4a95-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598697 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598527 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.598813 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598751 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43233bc0-0078-446d-b670-b8c8159a4a95-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599033 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599008 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599019 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599085 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599115 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.598824 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599416 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599442 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43233bc0-0078-446d-b670-b8c8159a4a95-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.599519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.599453 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.600901 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.600880 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43233bc0-0078-446d-b670-b8c8159a4a95-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.601474 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.601457 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43233bc0-0078-446d-b670-b8c8159a4a95-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.607177 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.607157 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr727\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-kube-api-access-rr727\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.607962 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.607940 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43233bc0-0078-446d-b670-b8c8159a4a95-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-zd2ql\" (UID: \"43233bc0-0078-446d-b670-b8c8159a4a95\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.725567 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.725484 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:07.854794 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.854771 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql"] Apr 24 16:52:07.859941 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:52:07.859909 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43233bc0_0078_446d_b670_b8c8159a4a95.slice/crio-c7595d823645448538ffeae978fb0ff6bb87051ff7ef759eec8eab8279c6de38 WatchSource:0}: Error finding container c7595d823645448538ffeae978fb0ff6bb87051ff7ef759eec8eab8279c6de38: Status 404 returned error can't find the container with id c7595d823645448538ffeae978fb0ff6bb87051ff7ef759eec8eab8279c6de38 Apr 24 16:52:07.862715 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.862686 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:52:07.862826 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.862765 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:52:07.862826 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:07.862807 2561 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Apr 24 16:52:08.247424 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:08.247385 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" event={"ID":"43233bc0-0078-446d-b670-b8c8159a4a95","Type":"ContainerStarted","Data":"d80e56a6e6f4317669ce010acce1dff5b90d533904b8ab98373c112f13a7739f"} Apr 24 16:52:08.247424 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:08.247429 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" event={"ID":"43233bc0-0078-446d-b670-b8c8159a4a95","Type":"ContainerStarted","Data":"c7595d823645448538ffeae978fb0ff6bb87051ff7ef759eec8eab8279c6de38"} Apr 24 16:52:08.268833 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:08.268785 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" podStartSLOduration=1.268771057 podStartE2EDuration="1.268771057s" podCreationTimestamp="2026-04-24 16:52:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:52:08.267422075 +0000 UTC m=+769.016592837" watchObservedRunningTime="2026-04-24 16:52:08.268771057 +0000 UTC m=+769.017941817" Apr 24 16:52:08.726310 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:08.726275 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:08.731398 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:08.731377 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:09.251243 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:09.251213 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:09.252120 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:09.252102 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-zd2ql" Apr 24 16:52:25.823600 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.823572 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:25.827224 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.827204 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.829917 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.829893 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Apr 24 16:52:25.830706 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.830692 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q89jh\"" Apr 24 16:52:25.837213 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.837189 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:25.958239 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958416 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958416 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958416 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958416 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958615 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:25.958615 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:25.958487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhqh\" (UniqueName: \"kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.059935 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.059896 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060106 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060065 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhqh\" (UniqueName: \"kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060168 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060125 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060229 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060202 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060276 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060240 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060325 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060286 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060325 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060315 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060325 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060318 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060466 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060382 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060597 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060573 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.060687 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.060648 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.062407 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.062385 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.062573 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.062556 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.067810 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.067790 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhqh\" (UniqueName: \"kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.138753 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.138721 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:26.282715 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.282691 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:26.284967 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:52:26.284926 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc011679_3f5d_4c4c_9b26_ca60e0ef7398.slice/crio-250109fc412a5e54095e32c2a140feb279c7dd534c9aa7513f479a03dd5e1184 WatchSource:0}: Error finding container 250109fc412a5e54095e32c2a140feb279c7dd534c9aa7513f479a03dd5e1184: Status 404 returned error can't find the container with id 250109fc412a5e54095e32c2a140feb279c7dd534c9aa7513f479a03dd5e1184 Apr 24 16:52:26.308306 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:26.308278 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerStarted","Data":"250109fc412a5e54095e32c2a140feb279c7dd534c9aa7513f479a03dd5e1184"} Apr 24 16:52:30.331049 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:30.330969 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerStarted","Data":"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426"} Apr 24 16:52:34.347339 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:34.347258 2561 generic.go:358] "Generic (PLEG): container finished" podID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerID="7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426" exitCode=0 Apr 24 16:52:34.347720 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:34.347332 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerDied","Data":"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426"} Apr 24 16:52:34.348469 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:34.348444 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 16:52:36.356229 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:36.356197 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerStarted","Data":"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748"} Apr 24 16:52:36.378654 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:36.378604 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" podStartSLOduration=2.202685871 podStartE2EDuration="11.37858898s" podCreationTimestamp="2026-04-24 16:52:25 +0000 UTC" firstStartedPulling="2026-04-24 16:52:26.28748407 +0000 UTC m=+787.036654810" lastFinishedPulling="2026-04-24 16:52:35.463387177 +0000 UTC m=+796.212557919" observedRunningTime="2026-04-24 16:52:36.376277386 +0000 UTC m=+797.125448140" watchObservedRunningTime="2026-04-24 16:52:36.37858898 +0000 UTC m=+797.127759739" Apr 24 16:52:45.448275 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.448240 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:45.448688 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.448603 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="main" containerID="cri-o://16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748" gracePeriod=30 Apr 24 16:52:45.707172 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.707122 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:45.843659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843629 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843659 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843662 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843705 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843729 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843763 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843820 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnhqh\" (UniqueName: \"kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.843899 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.843887 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home\") pod \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\" (UID: \"bc011679-3f5d-4c4c-9b26-ca60e0ef7398\") " Apr 24 16:52:45.844129 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.844009 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:52:45.844129 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.844027 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:52:45.844260 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.844242 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home" (OuterVolumeSpecName: "home") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:52:45.844324 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.844269 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.844324 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.844288 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.846147 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.846117 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm" (OuterVolumeSpecName: "dshm") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:52:45.846283 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.846145 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh" (OuterVolumeSpecName: "kube-api-access-mnhqh") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "kube-api-access-mnhqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:52:45.846283 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.846209 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:52:45.906128 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.906066 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc011679-3f5d-4c4c-9b26-ca60e0ef7398" (UID: "bc011679-3f5d-4c4c-9b26-ca60e0ef7398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:52:45.944862 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.944835 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.944862 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.944861 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.944862 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.944871 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.945046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.944881 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:45.945046 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:45.944891 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhqh\" (UniqueName: \"kubernetes.io/projected/bc011679-3f5d-4c4c-9b26-ca60e0ef7398-kube-api-access-mnhqh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:52:46.387979 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.387948 2561 generic.go:358] "Generic (PLEG): container finished" podID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerID="16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748" exitCode=0 Apr 24 16:52:46.388146 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.388017 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" Apr 24 16:52:46.388146 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.388016 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerDied","Data":"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748"} Apr 24 16:52:46.388146 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.388120 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv" event={"ID":"bc011679-3f5d-4c4c-9b26-ca60e0ef7398","Type":"ContainerDied","Data":"250109fc412a5e54095e32c2a140feb279c7dd534c9aa7513f479a03dd5e1184"} Apr 24 16:52:46.388146 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.388135 2561 scope.go:117] "RemoveContainer" containerID="16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748" Apr 24 16:52:46.396519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.396503 2561 scope.go:117] "RemoveContainer" containerID="7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426" Apr 24 16:52:46.412633 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.412607 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:46.416004 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.415979 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7dd9485f-kmzxv"] Apr 24 16:52:46.853754 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.853480 2561 scope.go:117] "RemoveContainer" containerID="16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748" Apr 24 16:52:46.854027 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:52:46.853838 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748\": container with ID starting with 16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748 not found: ID does not exist" containerID="16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748" Apr 24 16:52:46.854027 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.853865 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748"} err="failed to get container status \"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748\": rpc error: code = NotFound desc = could not find container \"16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748\": container with ID starting with 16c37ef27956bb2ff25a0e25aeb6fea4b0f17d3f6b6ec3d1d7c3cd0d0b8b3748 not found: ID does not exist" Apr 24 16:52:46.854027 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.853886 2561 scope.go:117] "RemoveContainer" containerID="7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426" Apr 24 16:52:46.854159 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:52:46.854121 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426\": container with ID starting with 7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426 not found: ID does not exist" containerID="7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426" Apr 24 16:52:46.854159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:46.854136 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426"} err="failed to get container status \"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426\": rpc error: code = NotFound desc = could not find container \"7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426\": container with ID starting with 7dbe61f8e6801a153274973feb0dd95e14e0e6774e8382df37b551d910383426 not found: ID does not exist" Apr 24 16:52:47.851116 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:52:47.851079 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" path="/var/lib/kubelet/pods/bc011679-3f5d-4c4c-9b26-ca60e0ef7398/volumes" Apr 24 16:53:09.907934 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.907903 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:53:09.908380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.908243 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="storage-initializer" Apr 24 16:53:09.908380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.908254 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="storage-initializer" Apr 24 16:53:09.908380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.908262 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="main" Apr 24 16:53:09.908380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.908267 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="main" Apr 24 16:53:09.908380 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.908324 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc011679-3f5d-4c4c-9b26-ca60e0ef7398" containerName="main" Apr 24 16:53:09.911368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.911348 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.914443 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.914418 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\"" Apr 24 16:53:09.915743 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.915652 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q89jh\"" Apr 24 16:53:09.926415 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.926391 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:53:09.948517 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948627 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948529 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948627 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948627 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948802 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948802 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948729 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:09.948802 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:09.948794 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6csd\" (UniqueName: \"kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.049814 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049780 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6csd\" (UniqueName: \"kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049880 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049897 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049917 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049935 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049957 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050001 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.049972 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050407 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.050375 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050407 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.050398 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050558 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.050491 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.050640 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.050622 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.052272 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.052253 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.052428 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.052411 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.058159 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.058138 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6csd\" (UniqueName: \"kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.223335 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.223253 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:53:10.353863 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.353836 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:53:10.355122 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:53:10.355095 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a934dd1_37db_4e37_afda_f9bff05ee2de.slice/crio-2fddf8201640b947f382f242ae40ae2e7db3521945db33db1d2af2f193f54416 WatchSource:0}: Error finding container 2fddf8201640b947f382f242ae40ae2e7db3521945db33db1d2af2f193f54416: Status 404 returned error can't find the container with id 2fddf8201640b947f382f242ae40ae2e7db3521945db33db1d2af2f193f54416 Apr 24 16:53:10.472643 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.472604 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerStarted","Data":"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6"} Apr 24 16:53:10.472821 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:53:10.472651 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerStarted","Data":"2fddf8201640b947f382f242ae40ae2e7db3521945db33db1d2af2f193f54416"} Apr 24 16:54:19.819418 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:19.819351 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:54:19.821055 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:19.821033 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:54:52.807710 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:52.807658 2561 generic.go:358] "Generic (PLEG): container finished" podID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerID="93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6" exitCode=0 Apr 24 16:54:52.808133 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:52.807733 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerDied","Data":"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6"} Apr 24 16:54:53.813496 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:53.813458 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerStarted","Data":"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263"} Apr 24 16:54:53.834333 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:54:53.834281 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" podStartSLOduration=104.834268331 podStartE2EDuration="1m44.834268331s" podCreationTimestamp="2026-04-24 16:53:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:54:53.831819732 +0000 UTC m=+934.580990504" watchObservedRunningTime="2026-04-24 16:54:53.834268331 +0000 UTC m=+934.583439091" Apr 24 16:55:00.224269 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:00.224232 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:00.224269 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:00.224277 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:00.236456 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:00.236432 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:00.849441 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:00.849410 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:02.138566 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:02.138534 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:55:02.844689 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:02.844615 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="main" containerID="cri-o://9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263" gracePeriod=30 Apr 24 16:55:03.090190 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.090167 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:03.249916 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.249890 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.249933 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6csd\" (UniqueName: \"kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.249971 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.249990 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250013 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250031 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250054 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache\") pod \"0a934dd1-37db-4e37-afda-f9bff05ee2de\" (UID: \"0a934dd1-37db-4e37-afda-f9bff05ee2de\") " Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250304 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:55:03.250337 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250316 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home" (OuterVolumeSpecName: "home") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:55:03.250623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.250452 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:55:03.252094 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.252068 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd" (OuterVolumeSpecName: "kube-api-access-k6csd") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "kube-api-access-k6csd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:55:03.252362 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.252337 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm" (OuterVolumeSpecName: "dshm") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:55:03.252448 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.252430 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:55:03.309170 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.309137 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a934dd1-37db-4e37-afda-f9bff05ee2de" (UID: "0a934dd1-37db-4e37-afda-f9bff05ee2de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:55:03.351592 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351570 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351592 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351591 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6csd\" (UniqueName: \"kubernetes.io/projected/0a934dd1-37db-4e37-afda-f9bff05ee2de-kube-api-access-k6csd\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351601 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a934dd1-37db-4e37-afda-f9bff05ee2de-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351611 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351619 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351627 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.351752 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.351636 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a934dd1-37db-4e37-afda-f9bff05ee2de-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:55:03.849074 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.849040 2561 generic.go:358] "Generic (PLEG): container finished" podID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerID="9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263" exitCode=0 Apr 24 16:55:03.849250 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.849134 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" Apr 24 16:55:03.849971 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.849951 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerDied","Data":"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263"} Apr 24 16:55:03.850051 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.849978 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb" event={"ID":"0a934dd1-37db-4e37-afda-f9bff05ee2de","Type":"ContainerDied","Data":"2fddf8201640b947f382f242ae40ae2e7db3521945db33db1d2af2f193f54416"} Apr 24 16:55:03.850051 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.849993 2561 scope.go:117] "RemoveContainer" containerID="9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263" Apr 24 16:55:03.859157 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.859136 2561 scope.go:117] "RemoveContainer" containerID="93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6" Apr 24 16:55:03.874540 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.874516 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:55:03.876053 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.876029 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-65c885996dmskfb"] Apr 24 16:55:03.923806 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.923787 2561 scope.go:117] "RemoveContainer" containerID="9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263" Apr 24 16:55:03.924137 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:55:03.924115 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263\": container with ID starting with 9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263 not found: ID does not exist" containerID="9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263" Apr 24 16:55:03.924192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.924147 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263"} err="failed to get container status \"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263\": rpc error: code = NotFound desc = could not find container \"9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263\": container with ID starting with 9713625109f1ced1fdde32c8506450d956b65997dbb7661d024e19f49d608263 not found: ID does not exist" Apr 24 16:55:03.924192 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.924168 2561 scope.go:117] "RemoveContainer" containerID="93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6" Apr 24 16:55:03.924455 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:55:03.924439 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6\": container with ID starting with 93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6 not found: ID does not exist" containerID="93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6" Apr 24 16:55:03.924509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:03.924462 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6"} err="failed to get container status \"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6\": rpc error: code = NotFound desc = could not find container \"93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6\": container with ID starting with 93e9c5d025bc87d1ee86ac814c9e1c9085dddd9e75ed2c207cfe805ec33c4de6 not found: ID does not exist" Apr 24 16:55:05.850236 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:05.850205 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" path="/var/lib/kubelet/pods/0a934dd1-37db-4e37-afda-f9bff05ee2de/volumes" Apr 24 16:55:15.716151 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716118 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:55:15.716577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716464 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="storage-initializer" Apr 24 16:55:15.716577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716474 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="storage-initializer" Apr 24 16:55:15.716577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716488 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="main" Apr 24 16:55:15.716577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716494 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="main" Apr 24 16:55:15.716577 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.716556 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="0a934dd1-37db-4e37-afda-f9bff05ee2de" containerName="main" Apr 24 16:55:15.718604 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.718587 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.720961 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.720940 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\"" Apr 24 16:55:15.721066 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.721051 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q89jh\"" Apr 24 16:55:15.730264 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.730242 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:55:15.756386 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756496 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756496 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756453 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756496 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756596 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756504 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756596 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756524 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2b5\" (UniqueName: \"kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.756596 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.756542 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.857870 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.857833 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858064 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.857904 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858253 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858138 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858253 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858186 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858250 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858360 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858388 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858425 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2b5\" (UniqueName: \"kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858459 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858823 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858465 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.858823 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.858645 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.860340 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.860320 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.860654 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.860635 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:15.866335 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:15.866308 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2b5\" (UniqueName: \"kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:16.029367 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:16.029301 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:55:16.157455 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:16.157428 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:55:16.159677 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:55:16.159633 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2126d62_20af_4d2c_af45_9b11e1b12687.slice/crio-f6d29ef5e7607b11b14143bf7d10d1567d7c3dc7f2e483b33f224a7efc6d7943 WatchSource:0}: Error finding container f6d29ef5e7607b11b14143bf7d10d1567d7c3dc7f2e483b33f224a7efc6d7943: Status 404 returned error can't find the container with id f6d29ef5e7607b11b14143bf7d10d1567d7c3dc7f2e483b33f224a7efc6d7943 Apr 24 16:55:16.892492 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:16.892449 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerStarted","Data":"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0"} Apr 24 16:55:16.893094 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:16.892505 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerStarted","Data":"f6d29ef5e7607b11b14143bf7d10d1567d7c3dc7f2e483b33f224a7efc6d7943"} Apr 24 16:55:36.958461 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:36.958429 2561 generic.go:358] "Generic (PLEG): container finished" podID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerID="2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0" exitCode=0 Apr 24 16:55:36.958863 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:55:36.958507 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerDied","Data":"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0"} Apr 24 16:56:04.073590 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:04.073509 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerStarted","Data":"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a"} Apr 24 16:56:04.095353 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:04.095296 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podStartSLOduration=22.433360109 podStartE2EDuration="49.095277989s" podCreationTimestamp="2026-04-24 16:55:15 +0000 UTC" firstStartedPulling="2026-04-24 16:55:36.959600729 +0000 UTC m=+977.708771472" lastFinishedPulling="2026-04-24 16:56:03.6215186 +0000 UTC m=+1004.370689352" observedRunningTime="2026-04-24 16:56:04.093825294 +0000 UTC m=+1004.842996066" watchObservedRunningTime="2026-04-24 16:56:04.095277989 +0000 UTC m=+1004.844448750" Apr 24 16:56:06.030376 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:06.030337 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:56:06.030833 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:06.030659 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:56:06.031877 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:06.031851 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:56:16.029787 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:16.029737 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:56:26.030401 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:26.030363 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:56:36.030444 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:36.030401 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:56:46.030417 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:46.030373 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:56:56.030509 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:56:56.030472 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:57:06.029982 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:06.029905 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:57:16.029985 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:16.029942 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:57:26.030372 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:26.030328 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused" Apr 24 16:57:36.039558 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:36.039530 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:57:36.047003 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:36.046975 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:57:41.842945 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:41.842899 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:57:41.843425 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:57:41.843234 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" containerID="cri-o://2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a" gracePeriod=30 Apr 24 16:58:00.328219 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.328184 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 16:58:00.333319 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.333298 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.336266 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.336245 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Apr 24 16:58:00.345722 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.345699 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 16:58:00.396030 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.395996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6bx\" (UniqueName: \"kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396174 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396172 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.396330 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.396208 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.497776 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497739 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.497956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497791 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.497956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497822 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6bx\" (UniqueName: \"kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.497956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497853 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.497956 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497910 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498168 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.497968 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498168 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.498064 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498350 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.498322 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498507 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.498476 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498591 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.498509 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.498591 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.498522 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.500421 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.500400 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.500570 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.500553 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.506974 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.506951 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6bx\" (UniqueName: \"kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx\") pod \"custom-route-timeout-test-kserve-578c565748-wnmhl\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.643087 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.643056 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:00.776304 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.776270 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 16:58:00.780001 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:58:00.779968 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc4177a_70d1_40d8_a65c_0ee79a0c27b5.slice/crio-0f3e6fa9a7b55c5536c4cbb56d509a767f5a26d216bd62fe020520dc3f0b3829 WatchSource:0}: Error finding container 0f3e6fa9a7b55c5536c4cbb56d509a767f5a26d216bd62fe020520dc3f0b3829: Status 404 returned error can't find the container with id 0f3e6fa9a7b55c5536c4cbb56d509a767f5a26d216bd62fe020520dc3f0b3829 Apr 24 16:58:00.781631 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:00.781616 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 16:58:01.483876 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:01.483840 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerStarted","Data":"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8"} Apr 24 16:58:01.483876 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:01.483881 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerStarted","Data":"0f3e6fa9a7b55c5536c4cbb56d509a767f5a26d216bd62fe020520dc3f0b3829"} Apr 24 16:58:05.499432 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:05.499396 2561 generic.go:358] "Generic (PLEG): container finished" podID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerID="25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8" exitCode=0 Apr 24 16:58:05.499822 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:05.499475 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerDied","Data":"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8"} Apr 24 16:58:06.504450 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:06.504417 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerStarted","Data":"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084"} Apr 24 16:58:06.527522 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:06.527468 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podStartSLOduration=6.527452218 podStartE2EDuration="6.527452218s" podCreationTimestamp="2026-04-24 16:58:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 16:58:06.525328998 +0000 UTC m=+1127.274499758" watchObservedRunningTime="2026-04-24 16:58:06.527452218 +0000 UTC m=+1127.276622978" Apr 24 16:58:10.643995 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:10.643955 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:10.644464 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:10.644003 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:58:10.645662 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:10.645632 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:58:12.083727 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.083698 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9_f2126d62-20af-4d2c-af45-9b11e1b12687/main/0.log" Apr 24 16:58:12.084114 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.084027 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:58:12.200640 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200585 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.200857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200663 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.200857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200738 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x2b5\" (UniqueName: \"kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.200857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200776 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.200857 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200813 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.201084 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.200864 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:58:12.201251 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.201225 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.201372 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.201270 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home\") pod \"f2126d62-20af-4d2c-af45-9b11e1b12687\" (UID: \"f2126d62-20af-4d2c-af45-9b11e1b12687\") " Apr 24 16:58:12.201770 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.201729 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.202182 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.202149 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home" (OuterVolumeSpecName: "home") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:58:12.203351 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.203146 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm" (OuterVolumeSpecName: "dshm") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:58:12.203555 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.203516 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 16:58:12.203854 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.203826 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5" (OuterVolumeSpecName: "kube-api-access-5x2b5") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "kube-api-access-5x2b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 16:58:12.212153 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.212123 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:58:12.254612 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.254569 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2126d62-20af-4d2c-af45-9b11e1b12687" (UID: "f2126d62-20af-4d2c-af45-9b11e1b12687"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 16:58:12.302519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302481 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x2b5\" (UniqueName: \"kubernetes.io/projected/f2126d62-20af-4d2c-af45-9b11e1b12687-kube-api-access-5x2b5\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.302519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302511 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.302519 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302520 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2126d62-20af-4d2c-af45-9b11e1b12687-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.302745 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302529 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.302745 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302539 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.302745 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.302548 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2126d62-20af-4d2c-af45-9b11e1b12687-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 16:58:12.526929 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.526854 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9_f2126d62-20af-4d2c-af45-9b11e1b12687/main/0.log" Apr 24 16:58:12.527279 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.527248 2561 generic.go:358] "Generic (PLEG): container finished" podID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerID="2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a" exitCode=137 Apr 24 16:58:12.527390 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.527320 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" Apr 24 16:58:12.527390 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.527374 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerDied","Data":"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a"} Apr 24 16:58:12.527484 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.527436 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9" event={"ID":"f2126d62-20af-4d2c-af45-9b11e1b12687","Type":"ContainerDied","Data":"f6d29ef5e7607b11b14143bf7d10d1567d7c3dc7f2e483b33f224a7efc6d7943"} Apr 24 16:58:12.527484 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.527457 2561 scope.go:117] "RemoveContainer" containerID="2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a" Apr 24 16:58:12.542742 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.542721 2561 scope.go:117] "RemoveContainer" containerID="2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0" Apr 24 16:58:12.555865 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.555837 2561 scope.go:117] "RemoveContainer" containerID="2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a" Apr 24 16:58:12.556175 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:58:12.556145 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a\": container with ID starting with 2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a not found: ID does not exist" containerID="2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a" Apr 24 16:58:12.556275 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.556185 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a"} err="failed to get container status \"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a\": rpc error: code = NotFound desc = could not find container \"2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a\": container with ID starting with 2b7421ef785289664997c8fadff9d76c23360c8d461705a6cbb50e5e72213b2a not found: ID does not exist" Apr 24 16:58:12.556275 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.556209 2561 scope.go:117] "RemoveContainer" containerID="2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0" Apr 24 16:58:12.556275 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.556224 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:58:12.556493 ip-10-0-139-51 kubenswrapper[2561]: E0424 16:58:12.556474 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0\": container with ID starting with 2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0 not found: ID does not exist" containerID="2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0" Apr 24 16:58:12.556530 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.556502 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0"} err="failed to get container status \"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0\": rpc error: code = NotFound desc = could not find container \"2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0\": container with ID starting with 2723675f9d67817a12702f3dfe4331cadc1a4d807f94e01031b8f2777a48c1c0 not found: ID does not exist" Apr 24 16:58:12.568925 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:12.568902 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6f95c5c56-t8gw9"] Apr 24 16:58:13.850643 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:13.850608 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" path="/var/lib/kubelet/pods/f2126d62-20af-4d2c-af45-9b11e1b12687/volumes" Apr 24 16:58:20.643510 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:20.643469 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:58:30.644270 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:30.644225 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:58:31.851243 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851214 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 16:58:31.851588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851551 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" Apr 24 16:58:31.851588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851563 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" Apr 24 16:58:31.851692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851592 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="storage-initializer" Apr 24 16:58:31.851692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851597 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="storage-initializer" Apr 24 16:58:31.851692 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.851645 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2126d62-20af-4d2c-af45-9b11e1b12687" containerName="main" Apr 24 16:58:31.857240 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.857224 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.859837 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.859815 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\"" Apr 24 16:58:31.866718 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.866694 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 16:58:31.979368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979337 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979368 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979625 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnktv\" (UniqueName: \"kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979625 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979499 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979625 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979587 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979625 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979620 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:31.979861 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:31.979740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.080865 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.080832 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081066 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.080881 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081066 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.080906 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081066 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081021 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nnktv\" (UniqueName: \"kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081066 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081063 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081129 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081300 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081152 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081406 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081293 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081485 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081458 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081540 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081513 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.081623 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.081604 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.083290 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.083269 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.083637 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.083615 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.091253 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.091230 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnktv\" (UniqueName: \"kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv\") pod \"precise-prefix-cache-test-kserve-646bd5b44b-sb2f6\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.169498 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.169461 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 16:58:32.304453 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.304426 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 16:58:32.306546 ip-10-0-139-51 kubenswrapper[2561]: W0424 16:58:32.306518 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bc3cbf_4956_4618_82dd_d1e5d8fb106d.slice/crio-46f5863be45cbbb07a8466432f496b511bbbbec2d9dede6566ffc0c7813b4a5a WatchSource:0}: Error finding container 46f5863be45cbbb07a8466432f496b511bbbbec2d9dede6566ffc0c7813b4a5a: Status 404 returned error can't find the container with id 46f5863be45cbbb07a8466432f496b511bbbbec2d9dede6566ffc0c7813b4a5a Apr 24 16:58:32.597894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.597783 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" event={"ID":"20bc3cbf-4956-4618-82dd-d1e5d8fb106d","Type":"ContainerStarted","Data":"6cb16a0cd2b86819286951d9b48dee8585bbd6636066adc0d342d3e9633eea57"} Apr 24 16:58:32.597894 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:32.597837 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" event={"ID":"20bc3cbf-4956-4618-82dd-d1e5d8fb106d","Type":"ContainerStarted","Data":"46f5863be45cbbb07a8466432f496b511bbbbec2d9dede6566ffc0c7813b4a5a"} Apr 24 16:58:40.643588 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:40.643547 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:58:50.644133 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:58:50.644090 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:59:00.644078 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:00.644025 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:59:10.644413 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:10.644359 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:59:19.845644 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:19.845606 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:59:19.847439 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:19.847419 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 16:59:20.644003 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:20.643965 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:59:30.644556 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:30.644501 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused" Apr 24 16:59:40.653473 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:40.653439 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:59:40.661402 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:40.661379 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 16:59:46.551536 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:46.551497 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 16:59:46.552136 ip-10-0-139-51 kubenswrapper[2561]: I0424 16:59:46.551875 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" containerID="cri-o://94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084" gracePeriod=30 Apr 24 17:00:00.534546 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.534504 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:00:00.538517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.538492 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.540988 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.540967 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\"" Apr 24 17:00:00.549760 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.549736 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:00:00.703944 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.703912 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704116 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.703949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7psn\" (UniqueName: \"kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704116 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.703974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704116 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.704004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704287 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.704129 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704287 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.704182 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.704392 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.704317 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805609 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805534 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805609 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805600 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805618 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805639 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7psn\" (UniqueName: \"kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805657 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805705 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.805847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.805758 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.806102 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.806079 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.806151 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.806137 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.806204 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.806184 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.806290 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.806270 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.807868 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.807848 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.808026 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.808011 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.813805 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.813784 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7psn\" (UniqueName: \"kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn\") pod \"router-with-refs-test-kserve-7dcd74ff8d-zjsdl\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.848202 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.848182 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:00.988586 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:00:00.988557 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf836a_b21d_4797_88f4_48b16fee97b9.slice/crio-2697808459eb8aae903b345d94c8be8c850f6dd5759d7cc2e42a34deac56befc WatchSource:0}: Error finding container 2697808459eb8aae903b345d94c8be8c850f6dd5759d7cc2e42a34deac56befc: Status 404 returned error can't find the container with id 2697808459eb8aae903b345d94c8be8c850f6dd5759d7cc2e42a34deac56befc Apr 24 17:00:00.989156 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:00.989123 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:00:01.911532 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:01.911492 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerStarted","Data":"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0"} Apr 24 17:00:01.911532 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:01.911530 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerStarted","Data":"2697808459eb8aae903b345d94c8be8c850f6dd5759d7cc2e42a34deac56befc"} Apr 24 17:00:05.927415 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:05.927384 2561 generic.go:358] "Generic (PLEG): container finished" podID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerID="e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0" exitCode=0 Apr 24 17:00:05.927794 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:05.927461 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerDied","Data":"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0"} Apr 24 17:00:06.933158 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:06.933123 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerStarted","Data":"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f"} Apr 24 17:00:06.959641 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:06.959584 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podStartSLOduration=6.959569096 podStartE2EDuration="6.959569096s" podCreationTimestamp="2026-04-24 17:00:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:00:06.957560859 +0000 UTC m=+1247.706731618" watchObservedRunningTime="2026-04-24 17:00:06.959569096 +0000 UTC m=+1247.708739857" Apr 24 17:00:10.849310 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:10.849282 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:10.849813 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:10.849322 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:00:10.850912 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:10.850884 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:00:16.842833 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.842803 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-578c565748-wnmhl_9dc4177a-70d1-40d8-a65c-0ee79a0c27b5/main/0.log" Apr 24 17:00:16.843247 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.843231 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 17:00:16.966485 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966455 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966485 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966488 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt6bx\" (UniqueName: \"kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966506 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966570 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966621 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966643 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.966747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966700 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location\") pod \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\" (UID: \"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5\") " Apr 24 17:00:16.967061 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.966809 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:00:16.967061 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.967026 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:16.967186 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.967161 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home" (OuterVolumeSpecName: "home") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:00:16.969227 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969125 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-578c565748-wnmhl_9dc4177a-70d1-40d8-a65c-0ee79a0c27b5/main/0.log" Apr 24 17:00:16.969362 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969328 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:00:16.969531 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969509 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm" (OuterVolumeSpecName: "dshm") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:00:16.969531 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969516 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx" (OuterVolumeSpecName: "kube-api-access-xt6bx") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "kube-api-access-xt6bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:00:16.969715 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969656 2561 generic.go:358] "Generic (PLEG): container finished" podID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerID="94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084" exitCode=137 Apr 24 17:00:16.969779 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969764 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" Apr 24 17:00:16.969779 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969771 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerDied","Data":"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084"} Apr 24 17:00:16.969873 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969802 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl" event={"ID":"9dc4177a-70d1-40d8-a65c-0ee79a0c27b5","Type":"ContainerDied","Data":"0f3e6fa9a7b55c5536c4cbb56d509a767f5a26d216bd62fe020520dc3f0b3829"} Apr 24 17:00:16.969873 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.969823 2561 scope.go:117] "RemoveContainer" containerID="94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084" Apr 24 17:00:16.983556 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.983520 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:00:16.987851 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:16.987826 2561 scope.go:117] "RemoveContainer" containerID="25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8" Apr 24 17:00:17.039853 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.039805 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" (UID: "9dc4177a-70d1-40d8-a65c-0ee79a0c27b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:00:17.063629 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.063606 2561 scope.go:117] "RemoveContainer" containerID="94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084" Apr 24 17:00:17.064015 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:00:17.063992 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084\": container with ID starting with 94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084 not found: ID does not exist" containerID="94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084" Apr 24 17:00:17.064107 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.064020 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084"} err="failed to get container status \"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084\": rpc error: code = NotFound desc = could not find container \"94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084\": container with ID starting with 94fa3929fa80e4f1189a75c5110fd72e2c3cdda8c25038deb1802f54c7a73084 not found: ID does not exist" Apr 24 17:00:17.064107 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.064039 2561 scope.go:117] "RemoveContainer" containerID="25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8" Apr 24 17:00:17.064357 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:00:17.064335 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8\": container with ID starting with 25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8 not found: ID does not exist" containerID="25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8" Apr 24 17:00:17.064398 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.064365 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8"} err="failed to get container status \"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8\": rpc error: code = NotFound desc = could not find container \"25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8\": container with ID starting with 25df79ca977061f307719ffd5a04cede643051e037253fdaf1200fe1e51acdb8 not found: ID does not exist" Apr 24 17:00:17.068039 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068011 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.068118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068046 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt6bx\" (UniqueName: \"kubernetes.io/projected/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kube-api-access-xt6bx\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.068118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068062 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.068118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068074 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.068118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068086 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.068118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.068100 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:00:17.335522 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.335488 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 17:00:17.343385 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.343354 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-578c565748-wnmhl"] Apr 24 17:00:17.850840 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:17.850799 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" path="/var/lib/kubelet/pods/9dc4177a-70d1-40d8-a65c-0ee79a0c27b5/volumes" Apr 24 17:00:20.849399 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:20.849359 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:00:30.849140 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:30.849106 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:00:40.849506 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:40.849463 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:00:50.849440 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:00:50.849397 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:01:00.849231 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:00.849185 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:01:10.849222 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:10.849178 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:01:20.848913 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:20.848860 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:01:30.849160 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:30.849111 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused" Apr 24 17:01:40.858773 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:40.858702 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:01:40.866220 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:40.866196 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:01:46.658504 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:46.658473 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:01:46.659037 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:46.658754 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" containerID="cri-o://922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f" gracePeriod=30 Apr 24 17:01:56.565737 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.565697 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:01:56.566179 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.566110 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" Apr 24 17:01:56.566179 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.566128 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" Apr 24 17:01:56.566179 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.566169 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="storage-initializer" Apr 24 17:01:56.566179 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.566179 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="storage-initializer" Apr 24 17:01:56.566327 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.566279 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dc4177a-70d1-40d8-a65c-0ee79a0c27b5" containerName="main" Apr 24 17:01:56.573018 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.570853 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.574796 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.574767 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\"" Apr 24 17:01:56.574957 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.574813 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-bbkh4\"" Apr 24 17:01:56.581881 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.581854 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:01:56.585956 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.585931 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:01:56.588653 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.588638 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.603014 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.602990 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:01:56.711289 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711262 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711428 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711294 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711428 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711315 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkl5\" (UniqueName: \"kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711428 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711375 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711428 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711438 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711473 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8z9w\" (UniqueName: \"kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711534 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711768 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711768 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711598 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.711768 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711625 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.711768 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.711647 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812386 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812358 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812549 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812395 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812549 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812503 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812549 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812544 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812733 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812572 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkl5\" (UniqueName: \"kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812733 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812627 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812733 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812689 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812733 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812722 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812762 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m8z9w\" (UniqueName: \"kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812795 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812798 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812821 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812856 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812892 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.812945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812918 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.813305 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.812949 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.813305 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813075 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.813305 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813101 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.813305 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813273 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.813517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813330 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.813517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813502 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.813627 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.813514 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.815052 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.815006 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.815358 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.815338 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.815581 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.815557 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.815704 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.815563 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.822702 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.822631 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkl5\" (UniqueName: \"kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.827420 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.827396 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8z9w\" (UniqueName: \"kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:56.887815 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.887789 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:56.898652 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:56.898630 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:01:57.028319 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:57.028162 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:01:57.030622 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:01:57.030595 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7e3352_b860_46e2_9ca5_7cb24ce352bd.slice/crio-57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f WatchSource:0}: Error finding container 57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f: Status 404 returned error can't find the container with id 57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f Apr 24 17:01:57.048867 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:57.048845 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:01:57.050766 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:01:57.050741 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc2a4c4_2dcd_4216_ba36_b8026170f678.slice/crio-9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6 WatchSource:0}: Error finding container 9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6: Status 404 returned error can't find the container with id 9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6 Apr 24 17:01:57.340542 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:57.340506 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerStarted","Data":"57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f"} Apr 24 17:01:57.342156 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:57.342090 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerStarted","Data":"f6bd6bd138f3c738efe065d53ace9bdf6d165f00a2404d93b373ecf1ac5ba11c"} Apr 24 17:01:57.342156 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:57.342122 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerStarted","Data":"9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6"} Apr 24 17:01:58.348114 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:58.348055 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerStarted","Data":"983553dad57e9e4d68e777b7939bcc042bc3ed1eedd1f8c197b7a3cd5a568ff3"} Apr 24 17:01:58.348861 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:58.348325 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:01:59.354456 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:01:59.354417 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerStarted","Data":"70dac07c287d754cd655724c0c088228bd64f0969420138513c85118bca72795"} Apr 24 17:02:02.365846 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:02.365813 2561 generic.go:358] "Generic (PLEG): container finished" podID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerID="f6bd6bd138f3c738efe065d53ace9bdf6d165f00a2404d93b373ecf1ac5ba11c" exitCode=0 Apr 24 17:02:02.366294 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:02.365885 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerDied","Data":"f6bd6bd138f3c738efe065d53ace9bdf6d165f00a2404d93b373ecf1ac5ba11c"} Apr 24 17:02:03.372194 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:03.372161 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerStarted","Data":"f80402384a62391fce18b45d74423db56177d45783309480f0fe5ae2b78f290c"} Apr 24 17:02:03.396179 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:03.396124 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podStartSLOduration=7.396104297 podStartE2EDuration="7.396104297s" podCreationTimestamp="2026-04-24 17:01:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:02:03.394936718 +0000 UTC m=+1364.144107529" watchObservedRunningTime="2026-04-24 17:02:03.396104297 +0000 UTC m=+1364.145275058" Apr 24 17:02:06.899922 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:06.899883 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:02:06.900291 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:06.899933 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:02:06.901373 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:06.901344 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:02:10.371292 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:10.371258 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:02:16.899753 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:16.899710 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:02:16.947299 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:16.947279 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7dcd74ff8d-zjsdl_26cf836a-b21d-4797-88f4-48b16fee97b9/main/0.log" Apr 24 17:02:16.947685 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:16.947656 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:02:17.008935 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.008906 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009113 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.008943 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7psn\" (UniqueName: \"kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009113 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.008979 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009113 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009020 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009113 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009096 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009323 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009124 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009323 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009159 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir\") pod \"26cf836a-b21d-4797-88f4-48b16fee97b9\" (UID: \"26cf836a-b21d-4797-88f4-48b16fee97b9\") " Apr 24 17:02:17.009323 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009173 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:02:17.009550 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009530 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.009971 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.009939 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home" (OuterVolumeSpecName: "home") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:02:17.011342 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.011317 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm" (OuterVolumeSpecName: "dshm") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:02:17.011476 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.011446 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:02:17.011807 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.011786 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn" (OuterVolumeSpecName: "kube-api-access-h7psn") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "kube-api-access-h7psn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:02:17.028147 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.028102 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:02:17.066056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.065991 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26cf836a-b21d-4797-88f4-48b16fee97b9" (UID: "26cf836a-b21d-4797-88f4-48b16fee97b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:02:17.110328 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110285 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.110328 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110323 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7psn\" (UniqueName: \"kubernetes.io/projected/26cf836a-b21d-4797-88f4-48b16fee97b9-kube-api-access-h7psn\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.110328 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110336 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26cf836a-b21d-4797-88f4-48b16fee97b9-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.110633 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110345 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.110633 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110353 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.110633 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.110361 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26cf836a-b21d-4797-88f4-48b16fee97b9-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:02:17.425087 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425056 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7dcd74ff8d-zjsdl_26cf836a-b21d-4797-88f4-48b16fee97b9/main/0.log" Apr 24 17:02:17.425424 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425401 2561 generic.go:358] "Generic (PLEG): container finished" podID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerID="922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f" exitCode=137 Apr 24 17:02:17.425505 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425488 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" Apr 24 17:02:17.425588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425484 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerDied","Data":"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f"} Apr 24 17:02:17.425641 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425606 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl" event={"ID":"26cf836a-b21d-4797-88f4-48b16fee97b9","Type":"ContainerDied","Data":"2697808459eb8aae903b345d94c8be8c850f6dd5759d7cc2e42a34deac56befc"} Apr 24 17:02:17.425641 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.425622 2561 scope.go:117] "RemoveContainer" containerID="922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f" Apr 24 17:02:17.434425 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.434398 2561 scope.go:117] "RemoveContainer" containerID="e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0" Apr 24 17:02:17.452067 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.450763 2561 scope.go:117] "RemoveContainer" containerID="922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f" Apr 24 17:02:17.459563 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.459534 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:02:17.462230 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:02:17.462205 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f\": container with ID starting with 922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f not found: ID does not exist" containerID="922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f" Apr 24 17:02:17.462403 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.462378 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f"} err="failed to get container status \"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f\": rpc error: code = NotFound desc = could not find container \"922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f\": container with ID starting with 922da1f2d88d0033cdeae259734e48c68cb85dd02dd9887d55bfe9206963951f not found: ID does not exist" Apr 24 17:02:17.462521 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.462497 2561 scope.go:117] "RemoveContainer" containerID="e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0" Apr 24 17:02:17.462902 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:02:17.462875 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0\": container with ID starting with e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0 not found: ID does not exist" containerID="e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0" Apr 24 17:02:17.463008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.462909 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0"} err="failed to get container status \"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0\": rpc error: code = NotFound desc = could not find container \"e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0\": container with ID starting with e137fa26c6a3103608b5fcbeb8a2e0588c1024d091eb511680ae2871c77a87e0 not found: ID does not exist" Apr 24 17:02:17.464660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.464635 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7dcd74ff8d-zjsdl"] Apr 24 17:02:17.851421 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:17.851340 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" path="/var/lib/kubelet/pods/26cf836a-b21d-4797-88f4-48b16fee97b9/volumes" Apr 24 17:02:26.899189 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:26.899146 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:02:36.900204 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:36.900165 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:02:46.899414 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:46.899367 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:02:56.899613 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:02:56.899564 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:03:06.899335 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:06.899249 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:03:16.900429 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:16.900382 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:03:26.899542 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:26.899492 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused" Apr 24 17:03:36.908701 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:36.908653 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:03:36.916254 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:36.916217 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:03:49.752065 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:49.752032 2561 generic.go:358] "Generic (PLEG): container finished" podID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerID="70dac07c287d754cd655724c0c088228bd64f0969420138513c85118bca72795" exitCode=0 Apr 24 17:03:49.752465 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:49.752105 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerDied","Data":"70dac07c287d754cd655724c0c088228bd64f0969420138513c85118bca72795"} Apr 24 17:03:49.753338 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:49.753322 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 17:03:50.758734 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:50.758700 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerStarted","Data":"b2f2ddbdbe4983e22cc66d47e4913c855529c92a57ebb3d98d374d810394ff7a"} Apr 24 17:03:50.784435 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:50.784371 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podStartSLOduration=114.023470947 podStartE2EDuration="1m54.784355865s" podCreationTimestamp="2026-04-24 17:01:56 +0000 UTC" firstStartedPulling="2026-04-24 17:01:57.032437466 +0000 UTC m=+1357.781608208" lastFinishedPulling="2026-04-24 17:01:57.793322383 +0000 UTC m=+1358.542493126" observedRunningTime="2026-04-24 17:03:50.781993566 +0000 UTC m=+1471.531164326" watchObservedRunningTime="2026-04-24 17:03:50.784355865 +0000 UTC m=+1471.533526626" Apr 24 17:03:56.888079 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:56.888035 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:03:56.888528 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:56.888091 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:03:56.888528 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:03:56.888402 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:06.888905 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:06.888857 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:16.888549 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:16.888495 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:19.873176 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:19.873144 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:04:19.875155 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:19.875129 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:04:26.888900 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:26.888853 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:36.888407 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:36.888308 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:46.889260 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:46.889215 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:04:56.888624 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:04:56.888565 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:05:06.888961 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:06.888914 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:05:16.888522 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:16.888476 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8001/health\": dial tcp 10.133.0.36:8001: connect: connection refused" Apr 24 17:05:26.902832 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:26.902795 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:05:26.914498 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:26.914473 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:05:34.972367 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:34.972334 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:05:34.972831 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:34.972638 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" containerID="cri-o://f80402384a62391fce18b45d74423db56177d45783309480f0fe5ae2b78f290c" gracePeriod=30 Apr 24 17:05:34.982520 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:34.982490 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:05:34.982855 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:34.982828 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" containerID="cri-o://b2f2ddbdbe4983e22cc66d47e4913c855529c92a57ebb3d98d374d810394ff7a" gracePeriod=30 Apr 24 17:05:42.974530 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.974491 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:05:42.975230 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.974941 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="storage-initializer" Apr 24 17:05:42.975230 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.974961 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="storage-initializer" Apr 24 17:05:42.975230 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.974991 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" Apr 24 17:05:42.975230 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.975000 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" Apr 24 17:05:42.975230 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.975111 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="26cf836a-b21d-4797-88f4-48b16fee97b9" containerName="main" Apr 24 17:05:42.977385 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.977370 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:42.979953 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.979931 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Apr 24 17:05:42.980146 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.980131 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-h7bwj\"" Apr 24 17:05:42.989840 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.989817 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:05:42.993812 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.993787 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:05:42.996972 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:42.996950 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.012190 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.012166 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:05:43.021886 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.021857 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022050 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.021896 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022050 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.021920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022050 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.021999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022191 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022191 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022097 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022191 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022191 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022165 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022339 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022251 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022339 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022339 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022311 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx45l\" (UniqueName: \"kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022463 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.022463 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqxc\" (UniqueName: \"kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.022463 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.022406 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.122854 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.122816 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.122875 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.122913 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.122938 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.122985 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123017 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123045 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vx45l\" (UniqueName: \"kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123078 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123125 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqxc\" (UniqueName: \"kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123159 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123189 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123223 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123263 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123293 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123328 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123793 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123372 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123793 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123401 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123793 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123644 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.123951 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123939 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.124009 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.123967 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.124118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.124095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.124175 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.124118 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.125494 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.125471 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.125748 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.125727 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.125869 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.125851 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.126234 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.126214 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.131558 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.131536 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqxc\" (UniqueName: \"kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc\") pod \"custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.132656 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.132637 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx45l\" (UniqueName: \"kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l\") pod \"custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.286465 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.286378 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:43.307522 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.307482 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:05:43.428169 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.428133 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:05:43.431805 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:05:43.431777 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89189e28_9a07_4cb1_885d_7ec3782eee5e.slice/crio-4ee12154cbe00d548c3ef9d97988c0539c1b78725bbd68c594751414f26485f9 WatchSource:0}: Error finding container 4ee12154cbe00d548c3ef9d97988c0539c1b78725bbd68c594751414f26485f9: Status 404 returned error can't find the container with id 4ee12154cbe00d548c3ef9d97988c0539c1b78725bbd68c594751414f26485f9 Apr 24 17:05:43.452703 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:43.452067 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:05:43.466323 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:05:43.466287 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a5d3e8_b025_496d_9afd_fae1c916af14.slice/crio-11032fd9f27207f11e8127f04855e335e470cdd9b396358e8834cc0e48eaf3ee WatchSource:0}: Error finding container 11032fd9f27207f11e8127f04855e335e470cdd9b396358e8834cc0e48eaf3ee: Status 404 returned error can't find the container with id 11032fd9f27207f11e8127f04855e335e470cdd9b396358e8834cc0e48eaf3ee Apr 24 17:05:44.171737 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:44.171705 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerStarted","Data":"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42"} Apr 24 17:05:44.171737 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:44.171743 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerStarted","Data":"11032fd9f27207f11e8127f04855e335e470cdd9b396358e8834cc0e48eaf3ee"} Apr 24 17:05:44.173098 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:44.173071 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerStarted","Data":"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9"} Apr 24 17:05:44.173098 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:44.173099 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerStarted","Data":"4ee12154cbe00d548c3ef9d97988c0539c1b78725bbd68c594751414f26485f9"} Apr 24 17:05:44.173275 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:44.173158 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:05:45.181453 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:45.181399 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerStarted","Data":"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea"} Apr 24 17:05:54.214786 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:54.214749 2561 generic.go:358] "Generic (PLEG): container finished" podID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerID="0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42" exitCode=0 Apr 24 17:05:54.215195 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:54.214823 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerDied","Data":"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42"} Apr 24 17:05:55.220917 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:55.220885 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerStarted","Data":"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf"} Apr 24 17:05:55.248165 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:55.246560 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podStartSLOduration=13.246542787 podStartE2EDuration="13.246542787s" podCreationTimestamp="2026-04-24 17:05:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:05:55.242849576 +0000 UTC m=+1595.992020337" watchObservedRunningTime="2026-04-24 17:05:55.246542787 +0000 UTC m=+1595.995713548" Apr 24 17:05:56.197314 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:05:56.197285 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:06:03.308158 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:03.308077 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:06:03.308158 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:03.308122 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:06:03.309582 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:03.309553 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:04.982929 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:04.982883 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="llm-d-routing-sidecar" containerID="cri-o://983553dad57e9e4d68e777b7939bcc042bc3ed1eedd1f8c197b7a3cd5a568ff3" gracePeriod=2 Apr 24 17:06:05.256592 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.256564 2561 generic.go:358] "Generic (PLEG): container finished" podID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerID="f80402384a62391fce18b45d74423db56177d45783309480f0fe5ae2b78f290c" exitCode=137 Apr 24 17:06:05.256732 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.256640 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerDied","Data":"f80402384a62391fce18b45d74423db56177d45783309480f0fe5ae2b78f290c"} Apr 24 17:06:05.256732 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.256719 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" event={"ID":"ebc2a4c4-2dcd-4216-ba36-b8026170f678","Type":"ContainerDied","Data":"9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6"} Apr 24 17:06:05.256910 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.256736 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9615e02d23fa7dcf51cddfe295d99b942be9500f08c11fb24b06caf0871bf2e6" Apr 24 17:06:05.258168 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258148 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586_6f7e3352-b860-46e2-9ca5-7cb24ce352bd/main/0.log" Apr 24 17:06:05.258704 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258659 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:06:05.258880 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258857 2561 generic.go:358] "Generic (PLEG): container finished" podID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerID="b2f2ddbdbe4983e22cc66d47e4913c855529c92a57ebb3d98d374d810394ff7a" exitCode=137 Apr 24 17:06:05.258954 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258881 2561 generic.go:358] "Generic (PLEG): container finished" podID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerID="983553dad57e9e4d68e777b7939bcc042bc3ed1eedd1f8c197b7a3cd5a568ff3" exitCode=0 Apr 24 17:06:05.258954 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258932 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerDied","Data":"b2f2ddbdbe4983e22cc66d47e4913c855529c92a57ebb3d98d374d810394ff7a"} Apr 24 17:06:05.259034 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258967 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerDied","Data":"983553dad57e9e4d68e777b7939bcc042bc3ed1eedd1f8c197b7a3cd5a568ff3"} Apr 24 17:06:05.259034 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258981 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" event={"ID":"6f7e3352-b860-46e2-9ca5-7cb24ce352bd","Type":"ContainerDied","Data":"57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f"} Apr 24 17:06:05.259034 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.258992 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57931a46ef2009c550bca19089a4157a7ce6533d97b26503b9b6494243205e2f" Apr 24 17:06:05.262019 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.262002 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586_6f7e3352-b860-46e2-9ca5-7cb24ce352bd/main/0.log" Apr 24 17:06:05.262689 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.262643 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:06:05.341830 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341795 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.341830 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341837 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341871 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341894 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341909 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8z9w\" (UniqueName: \"kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341936 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341959 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.341991 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342058 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342046 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfkl5\" (UniqueName: \"kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342072 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342094 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342113 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342150 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location\") pod \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\" (UID: \"6f7e3352-b860-46e2-9ca5-7cb24ce352bd\") " Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342171 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342214 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342262 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location\") pod \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\" (UID: \"ebc2a4c4-2dcd-4216-ba36-b8026170f678\") " Apr 24 17:06:05.342798 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342634 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.342798 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.342655 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.343861 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.343484 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home" (OuterVolumeSpecName: "home") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.344792 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.344553 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm" (OuterVolumeSpecName: "dshm") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.344792 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.344556 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm" (OuterVolumeSpecName: "dshm") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.344792 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.344741 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5" (OuterVolumeSpecName: "kube-api-access-qfkl5") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "kube-api-access-qfkl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:06:05.344792 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.344796 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:06:05.345629 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.345601 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home" (OuterVolumeSpecName: "home") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.346027 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.346005 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w" (OuterVolumeSpecName: "kube-api-access-m8z9w") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "kube-api-access-m8z9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:06:05.347524 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.347502 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:06:05.356447 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.356421 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.363178 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.363152 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.407896 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.407851 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc2a4c4-2dcd-4216-ba36-b8026170f678" (UID: "ebc2a4c4-2dcd-4216-ba36-b8026170f678"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.412225 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.412196 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f7e3352-b860-46e2-9ca5-7cb24ce352bd" (UID: "6f7e3352-b860-46e2-9ca5-7cb24ce352bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:06:05.443373 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443345 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkl5\" (UniqueName: \"kubernetes.io/projected/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kube-api-access-qfkl5\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443373 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443370 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443380 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443389 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443398 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc2a4c4-2dcd-4216-ba36-b8026170f678-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443409 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443418 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443426 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443434 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8z9w\" (UniqueName: \"kubernetes.io/projected/ebc2a4c4-2dcd-4216-ba36-b8026170f678-kube-api-access-m8z9w\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443443 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443451 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc2a4c4-2dcd-4216-ba36-b8026170f678-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:05.443507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:05.443458 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7e3352-b860-46e2-9ca5-7cb24ce352bd-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:06:06.263119 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.263035 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7" Apr 24 17:06:06.263502 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.263045 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586" Apr 24 17:06:06.282225 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.282188 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:06:06.285826 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.285795 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-58db7498998m586"] Apr 24 17:06:06.297214 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.297185 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:06:06.301056 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:06.301032 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7dgbkz7"] Apr 24 17:06:07.850251 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:07.850214 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" path="/var/lib/kubelet/pods/6f7e3352-b860-46e2-9ca5-7cb24ce352bd/volumes" Apr 24 17:06:07.850729 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:07.850714 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" path="/var/lib/kubelet/pods/ebc2a4c4-2dcd-4216-ba36-b8026170f678/volumes" Apr 24 17:06:13.308129 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:13.308086 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:23.308743 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:23.308691 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:33.308327 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:33.308280 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:43.308361 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:43.308316 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:53.308802 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:53.308763 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:06:57.448521 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:57.448487 2561 generic.go:358] "Generic (PLEG): container finished" podID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerID="e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea" exitCode=0 Apr 24 17:06:57.449012 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:57.448553 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerDied","Data":"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea"} Apr 24 17:06:58.454847 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:58.454803 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerStarted","Data":"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1"} Apr 24 17:06:58.479480 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:06:58.479422 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podStartSLOduration=76.479407729 podStartE2EDuration="1m16.479407729s" podCreationTimestamp="2026-04-24 17:05:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:06:58.476462687 +0000 UTC m=+1659.225633460" watchObservedRunningTime="2026-04-24 17:06:58.479407729 +0000 UTC m=+1659.228578489" Apr 24 17:07:03.287424 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:03.287388 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:07:03.287897 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:03.287440 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:07:03.287897 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:03.287815 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:07:03.308825 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:03.308784 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:07:13.286869 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:13.286820 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:07:13.307881 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:13.307840 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:07:23.287710 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:23.287646 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:07:23.308206 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:23.308167 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused" Apr 24 17:07:33.287462 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:33.287364 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:07:33.318064 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:33.318029 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:07:33.326454 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:33.326427 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:07:43.287862 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:43.287813 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:07:53.287606 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:07:53.287552 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:08:03.287121 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:03.287070 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:08:13.287119 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:13.287071 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:08:19.922125 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:19.922089 2561 scope.go:117] "RemoveContainer" containerID="70dac07c287d754cd655724c0c088228bd64f0969420138513c85118bca72795" Apr 24 17:08:19.981826 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:19.981802 2561 scope.go:117] "RemoveContainer" containerID="f6bd6bd138f3c738efe065d53ace9bdf6d165f00a2404d93b373ecf1ac5ba11c" Apr 24 17:08:20.039248 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:20.039225 2561 scope.go:117] "RemoveContainer" containerID="f80402384a62391fce18b45d74423db56177d45783309480f0fe5ae2b78f290c" Apr 24 17:08:20.046977 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:20.046960 2561 scope.go:117] "RemoveContainer" containerID="983553dad57e9e4d68e777b7939bcc042bc3ed1eedd1f8c197b7a3cd5a568ff3" Apr 24 17:08:23.287421 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:23.287375 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8001/health\": dial tcp 10.133.0.38:8001: connect: connection refused" Apr 24 17:08:33.296236 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:33.296200 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:08:33.307774 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:33.307753 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:08:41.617578 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:41.617548 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:08:41.618042 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:41.617933 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" containerID="cri-o://317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" gracePeriod=30 Apr 24 17:08:41.619510 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:41.619487 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:08:41.619789 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:41.619754 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" containerID="cri-o://261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf" gracePeriod=30 Apr 24 17:08:52.263562 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.263523 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:08:52.264031 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.263986 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="storage-initializer" Apr 24 17:08:52.264031 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264002 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="storage-initializer" Apr 24 17:08:52.264031 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264020 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" Apr 24 17:08:52.264031 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264026 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" Apr 24 17:08:52.264031 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264033 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264039 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264049 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="llm-d-routing-sidecar" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264054 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="llm-d-routing-sidecar" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264067 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="storage-initializer" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264072 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="storage-initializer" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264126 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="main" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264136 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ebc2a4c4-2dcd-4216-ba36-b8026170f678" containerName="main" Apr 24 17:08:52.264199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.264143 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6f7e3352-b860-46e2-9ca5-7cb24ce352bd" containerName="llm-d-routing-sidecar" Apr 24 17:08:52.267541 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.267525 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.270344 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.270320 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-wzxx6\"" Apr 24 17:08:52.270467 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.270377 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Apr 24 17:08:52.279314 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.279289 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:08:52.289125 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.289101 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:08:52.293066 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.293043 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.304900 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.304875 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:08:52.358848 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.358819 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.358859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.358885 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.358941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.358972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359008 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtz7j\" (UniqueName: \"kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359047 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359137 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zk6\" (UniqueName: \"kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359392 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.359392 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.359212 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.460546 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460514 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.460546 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460548 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460579 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtz7j\" (UniqueName: \"kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460610 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460645 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460687 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460709 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460737 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460759 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65zk6\" (UniqueName: \"kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.460800 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460791 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460816 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460916 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460951 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.460995 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461003 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461082 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.461226 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461108 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.461578 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461323 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461578 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461328 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.461578 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461349 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461578 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461470 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.461814 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.461604 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.463039 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.463012 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.463266 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.463251 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.463534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.463515 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.463574 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.463516 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.468232 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.468209 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtz7j\" (UniqueName: \"kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j\") pod \"router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.468328 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.468292 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zk6\" (UniqueName: \"kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6\") pod \"router-with-refs-pd-test-kserve-6b94f45878-fglj7\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.577403 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.577311 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:52.605135 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.605105 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:08:52.714116 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.714089 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:08:52.716302 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:08:52.716273 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dc5aca_d101_4c93_95c5_611d1543e22c.slice/crio-35ace01013b414b40480fe945f080fd45e242a585e56e8d4efef76d722a17ecf WatchSource:0}: Error finding container 35ace01013b414b40480fe945f080fd45e242a585e56e8d4efef76d722a17ecf: Status 404 returned error can't find the container with id 35ace01013b414b40480fe945f080fd45e242a585e56e8d4efef76d722a17ecf Apr 24 17:08:52.718373 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.718352 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 17:08:52.741904 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.741837 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:08:52.752798 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:08:52.752775 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b857573_4789_4ca6_9fe6_660c9283a66d.slice/crio-d96b2377e322f491e2e6285848149b8309ac6452e61e1c1a1b88c224b63dc490 WatchSource:0}: Error finding container d96b2377e322f491e2e6285848149b8309ac6452e61e1c1a1b88c224b63dc490: Status 404 returned error can't find the container with id d96b2377e322f491e2e6285848149b8309ac6452e61e1c1a1b88c224b63dc490 Apr 24 17:08:52.886085 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.886038 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerStarted","Data":"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806"} Apr 24 17:08:52.886085 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.886084 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerStarted","Data":"d96b2377e322f491e2e6285848149b8309ac6452e61e1c1a1b88c224b63dc490"} Apr 24 17:08:52.887722 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.887692 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerStarted","Data":"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87"} Apr 24 17:08:52.888155 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.887728 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerStarted","Data":"35ace01013b414b40480fe945f080fd45e242a585e56e8d4efef76d722a17ecf"} Apr 24 17:08:52.888155 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:52.887793 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:08:53.899621 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:08:53.899574 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerStarted","Data":"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802"} Apr 24 17:09:04.915142 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:04.915067 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:09:11.618456 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.618416 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="llm-d-routing-sidecar" containerID="cri-o://f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" gracePeriod=2 Apr 24 17:09:11.949770 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.949750 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn_89189e28-9a07-4cb1-885d-7ec3782eee5e/main/0.log" Apr 24 17:09:11.950393 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.950375 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:09:11.953125 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.953111 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:09:11.976092 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976069 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn_89189e28-9a07-4cb1-885d-7ec3782eee5e/main/0.log" Apr 24 17:09:11.976775 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976749 2561 generic.go:358] "Generic (PLEG): container finished" podID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerID="317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" exitCode=137 Apr 24 17:09:11.976775 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976772 2561 generic.go:358] "Generic (PLEG): container finished" podID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerID="f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" exitCode=0 Apr 24 17:09:11.976928 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976825 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" Apr 24 17:09:11.976928 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976831 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerDied","Data":"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1"} Apr 24 17:09:11.976928 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976875 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerDied","Data":"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9"} Apr 24 17:09:11.976928 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976885 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn" event={"ID":"89189e28-9a07-4cb1-885d-7ec3782eee5e","Type":"ContainerDied","Data":"4ee12154cbe00d548c3ef9d97988c0539c1b78725bbd68c594751414f26485f9"} Apr 24 17:09:11.976928 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.976900 2561 scope.go:117] "RemoveContainer" containerID="317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" Apr 24 17:09:11.978598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.978574 2561 generic.go:358] "Generic (PLEG): container finished" podID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerID="261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf" exitCode=137 Apr 24 17:09:11.978735 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.978607 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerDied","Data":"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf"} Apr 24 17:09:11.978735 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.978632 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" event={"ID":"88a5d3e8-b025-496d-9afd-fae1c916af14","Type":"ContainerDied","Data":"11032fd9f27207f11e8127f04855e335e470cdd9b396358e8834cc0e48eaf3ee"} Apr 24 17:09:11.978735 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.978643 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn" Apr 24 17:09:11.985185 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:11.985160 2561 scope.go:117] "RemoveContainer" containerID="e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea" Apr 24 17:09:12.045253 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.045230 2561 scope.go:117] "RemoveContainer" containerID="f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" Apr 24 17:09:12.052608 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.052592 2561 scope.go:117] "RemoveContainer" containerID="317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" Apr 24 17:09:12.052981 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:09:12.052957 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1\": container with ID starting with 317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1 not found: ID does not exist" containerID="317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" Apr 24 17:09:12.053074 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.052989 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1"} err="failed to get container status \"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1\": rpc error: code = NotFound desc = could not find container \"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1\": container with ID starting with 317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1 not found: ID does not exist" Apr 24 17:09:12.053074 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053025 2561 scope.go:117] "RemoveContainer" containerID="e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea" Apr 24 17:09:12.053290 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:09:12.053272 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea\": container with ID starting with e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea not found: ID does not exist" containerID="e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea" Apr 24 17:09:12.053339 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053300 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea"} err="failed to get container status \"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea\": rpc error: code = NotFound desc = could not find container \"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea\": container with ID starting with e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea not found: ID does not exist" Apr 24 17:09:12.053339 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053317 2561 scope.go:117] "RemoveContainer" containerID="f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" Apr 24 17:09:12.053531 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:09:12.053516 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9\": container with ID starting with f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9 not found: ID does not exist" containerID="f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" Apr 24 17:09:12.053569 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053537 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9"} err="failed to get container status \"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9\": rpc error: code = NotFound desc = could not find container \"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9\": container with ID starting with f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9 not found: ID does not exist" Apr 24 17:09:12.053569 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053550 2561 scope.go:117] "RemoveContainer" containerID="317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1" Apr 24 17:09:12.053799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053779 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1"} err="failed to get container status \"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1\": rpc error: code = NotFound desc = could not find container \"317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1\": container with ID starting with 317952b1cb87a88ce880079dae87640577599789481d63a881de396f12ddb9f1 not found: ID does not exist" Apr 24 17:09:12.053799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.053796 2561 scope.go:117] "RemoveContainer" containerID="e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea" Apr 24 17:09:12.054050 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.054034 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea"} err="failed to get container status \"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea\": rpc error: code = NotFound desc = could not find container \"e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea\": container with ID starting with e2724d81c03b8d3f104081d544aba34f9f111e1f86d192a60f189d00651846ea not found: ID does not exist" Apr 24 17:09:12.054091 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.054050 2561 scope.go:117] "RemoveContainer" containerID="f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9" Apr 24 17:09:12.054246 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.054223 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9"} err="failed to get container status \"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9\": rpc error: code = NotFound desc = could not find container \"f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9\": container with ID starting with f27fa1df4c1c9e0dc5198929c8566f29b33dfc0dc050aa95d393b4615edf2fc9 not found: ID does not exist" Apr 24 17:09:12.054299 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.054248 2561 scope.go:117] "RemoveContainer" containerID="261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf" Apr 24 17:09:12.055121 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055106 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055181 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055131 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055181 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055170 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055263 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055210 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055305 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055269 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055409 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055363 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055478 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055397 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055478 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055459 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055585 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055516 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bqxc\" (UniqueName: \"kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055585 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055566 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055709 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055601 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx45l\" (UniqueName: \"kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055709 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055626 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache\") pod \"88a5d3e8-b025-496d-9afd-fae1c916af14\" (UID: \"88a5d3e8-b025-496d-9afd-fae1c916af14\") " Apr 24 17:09:12.055709 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055685 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055873 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055725 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache\") pod \"89189e28-9a07-4cb1-885d-7ec3782eee5e\" (UID: \"89189e28-9a07-4cb1-885d-7ec3782eee5e\") " Apr 24 17:09:12.055873 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.055792 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home" (OuterVolumeSpecName: "home") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.056416 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.056100 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.056521 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.056417 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.056521 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.056425 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache" (OuterVolumeSpecName: "model-cache") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.056694 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.056649 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home" (OuterVolumeSpecName: "home") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.059416 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.059009 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:09:12.059416 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.059096 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc" (OuterVolumeSpecName: "kube-api-access-6bqxc") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "kube-api-access-6bqxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:09:12.059725 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.059625 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm" (OuterVolumeSpecName: "dshm") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.060221 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.060197 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l" (OuterVolumeSpecName: "kube-api-access-vx45l") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "kube-api-access-vx45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:09:12.060933 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.060907 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:09:12.061433 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.061401 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm" (OuterVolumeSpecName: "dshm") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.064884 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.064866 2561 scope.go:117] "RemoveContainer" containerID="0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42" Apr 24 17:09:12.071229 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.071203 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.073003 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.072980 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.120367 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.120274 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88a5d3e8-b025-496d-9afd-fae1c916af14" (UID: "88a5d3e8-b025-496d-9afd-fae1c916af14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.124516 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.124490 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89189e28-9a07-4cb1-885d-7ec3782eee5e" (UID: "89189e28-9a07-4cb1-885d-7ec3782eee5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:09:12.130620 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.130601 2561 scope.go:117] "RemoveContainer" containerID="261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf" Apr 24 17:09:12.130946 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:09:12.130926 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf\": container with ID starting with 261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf not found: ID does not exist" containerID="261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf" Apr 24 17:09:12.130997 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.130955 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf"} err="failed to get container status \"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf\": rpc error: code = NotFound desc = could not find container \"261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf\": container with ID starting with 261f51a0414d9428436451a30f5cb7d41e6fbd4c9dea893c97c4fa681afc40cf not found: ID does not exist" Apr 24 17:09:12.130997 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.130973 2561 scope.go:117] "RemoveContainer" containerID="0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42" Apr 24 17:09:12.131223 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:09:12.131208 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42\": container with ID starting with 0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42 not found: ID does not exist" containerID="0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42" Apr 24 17:09:12.131267 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.131226 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42"} err="failed to get container status \"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42\": rpc error: code = NotFound desc = could not find container \"0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42\": container with ID starting with 0cbf0de18fa70ebce52051a034fa29cbfbce4a1b3dd9a050442db0f267ac9f42 not found: ID does not exist" Apr 24 17:09:12.157166 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157143 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx45l\" (UniqueName: \"kubernetes.io/projected/88a5d3e8-b025-496d-9afd-fae1c916af14-kube-api-access-vx45l\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157166 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157165 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157174 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157182 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157193 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157201 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157209 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157218 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157226 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89189e28-9a07-4cb1-885d-7ec3782eee5e-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157233 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5d3e8-b025-496d-9afd-fae1c916af14-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157241 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89189e28-9a07-4cb1-885d-7ec3782eee5e-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157248 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqxc\" (UniqueName: \"kubernetes.io/projected/89189e28-9a07-4cb1-885d-7ec3782eee5e-kube-api-access-6bqxc\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.157295 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.157256 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5d3e8-b025-496d-9afd-fae1c916af14-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:09:12.301786 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.301760 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:09:12.305901 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.305878 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-789c4b6bd6-pq8pn"] Apr 24 17:09:12.317709 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.317684 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:09:12.321864 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:12.321846 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d4598fd77-x5cjn"] Apr 24 17:09:13.850095 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:13.850061 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" path="/var/lib/kubelet/pods/88a5d3e8-b025-496d-9afd-fae1c916af14/volumes" Apr 24 17:09:13.850610 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:13.850595 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" path="/var/lib/kubelet/pods/89189e28-9a07-4cb1-885d-7ec3782eee5e/volumes" Apr 24 17:09:19.897372 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:19.897344 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:09:19.900507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:19.900486 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:09:48.110348 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:48.110317 2561 generic.go:358] "Generic (PLEG): container finished" podID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerID="3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802" exitCode=0 Apr 24 17:09:48.110708 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:48.110389 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerDied","Data":"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802"} Apr 24 17:09:49.116726 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:49.116686 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerStarted","Data":"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5"} Apr 24 17:09:49.149121 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:49.149063 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podStartSLOduration=57.149044531 podStartE2EDuration="57.149044531s" podCreationTimestamp="2026-04-24 17:08:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:09:49.147711021 +0000 UTC m=+1829.896881784" watchObservedRunningTime="2026-04-24 17:09:49.149044531 +0000 UTC m=+1829.898215292" Apr 24 17:09:52.578049 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:52.578014 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:09:52.578049 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:52.578053 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:09:52.578550 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:09:52.578314 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:02.578712 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:02.578647 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:12.578176 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:12.578129 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:18.224337 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:18.224304 2561 generic.go:358] "Generic (PLEG): container finished" podID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerID="dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806" exitCode=0 Apr 24 17:10:18.224790 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:18.224343 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerDied","Data":"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806"} Apr 24 17:10:19.230353 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:19.230314 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerStarted","Data":"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8"} Apr 24 17:10:19.255644 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:19.255593 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podStartSLOduration=87.255578941 podStartE2EDuration="1m27.255578941s" podCreationTimestamp="2026-04-24 17:08:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:10:19.252680375 +0000 UTC m=+1860.001851130" watchObservedRunningTime="2026-04-24 17:10:19.255578941 +0000 UTC m=+1860.004749847" Apr 24 17:10:20.075883 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:20.075853 2561 scope.go:117] "RemoveContainer" containerID="b2f2ddbdbe4983e22cc66d47e4913c855529c92a57ebb3d98d374d810394ff7a" Apr 24 17:10:22.578504 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:22.578454 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:22.605458 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:22.605427 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:10:22.605650 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:22.605471 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:10:22.606703 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:22.606651 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:10:32.578425 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:32.578317 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:32.605637 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:32.605596 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:10:42.577692 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:42.577632 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:42.606415 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:42.606373 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:10:52.577790 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:52.577744 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:10:52.605455 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:10:52.605413 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:02.578475 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:02.578425 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:11:02.605787 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:02.605741 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:12.577853 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:12.577805 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:11:12.605803 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:12.605760 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:22.578455 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:22.578409 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused" Apr 24 17:11:22.605587 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:22.605548 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:32.593211 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:32.593181 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:11:32.605509 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:32.605467 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:32.605808 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:32.605789 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:11:42.605996 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:42.605952 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:11:52.605443 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:11:52.605399 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused" Apr 24 17:12:02.615034 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:02.614961 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:12:02.622632 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:02.622610 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:12:13.779234 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:13.779201 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:12:13.779718 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:13.779609 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" containerID="cri-o://4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" gracePeriod=30 Apr 24 17:12:13.789855 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:13.789830 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:12:13.790108 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:13.790086 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" containerID="cri-o://6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8" gracePeriod=30 Apr 24 17:12:32.835891 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.835854 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:12:32.836433 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836414 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" Apr 24 17:12:32.836481 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836437 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" Apr 24 17:12:32.836481 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836465 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="llm-d-routing-sidecar" Apr 24 17:12:32.836481 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836473 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="llm-d-routing-sidecar" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836489 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="storage-initializer" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836497 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="storage-initializer" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836511 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="storage-initializer" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836520 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="storage-initializer" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836533 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" Apr 24 17:12:32.836589 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836542 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" Apr 24 17:12:32.836788 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836618 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="llm-d-routing-sidecar" Apr 24 17:12:32.836788 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836628 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="89189e28-9a07-4cb1-885d-7ec3782eee5e" containerName="main" Apr 24 17:12:32.836788 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.836634 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="88a5d3e8-b025-496d-9afd-fae1c916af14" containerName="main" Apr 24 17:12:32.841209 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.841191 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.843489 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.843469 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\"" Apr 24 17:12:32.850732 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.850712 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:12:32.973924 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.973888 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcng\" (UniqueName: \"kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974144 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.973943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974144 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.974103 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974257 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.974143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974257 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.974171 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974257 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.974229 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:32.974377 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:32.974268 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075333 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075280 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075333 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075334 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075355 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcng\" (UniqueName: \"kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075388 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075486 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075534 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075562 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075874 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075790 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075874 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075829 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075946 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075902 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.075983 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.075956 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.077915 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.077892 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.078045 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.078028 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.084415 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.084391 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcng\" (UniqueName: \"kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.152710 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.152656 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:33.288507 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.288481 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:12:33.291089 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:12:33.291059 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251a216_8fc7_4bda_8bd0_a70dacd03003.slice/crio-c3cc2d91b9a0fef2d9ac7b742e29f6628ecbecb9705dc8e9c911d8a3875a0753 WatchSource:0}: Error finding container c3cc2d91b9a0fef2d9ac7b742e29f6628ecbecb9705dc8e9c911d8a3875a0753: Status 404 returned error can't find the container with id c3cc2d91b9a0fef2d9ac7b742e29f6628ecbecb9705dc8e9c911d8a3875a0753 Apr 24 17:12:33.727011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.726974 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerStarted","Data":"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405"} Apr 24 17:12:33.727011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:33.727019 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerStarted","Data":"c3cc2d91b9a0fef2d9ac7b742e29f6628ecbecb9705dc8e9c911d8a3875a0753"} Apr 24 17:12:37.742563 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:37.742530 2561 generic.go:358] "Generic (PLEG): container finished" podID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerID="94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405" exitCode=0 Apr 24 17:12:37.742969 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:37.742585 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerDied","Data":"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405"} Apr 24 17:12:38.748160 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:38.748125 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerStarted","Data":"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7"} Apr 24 17:12:38.772149 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:38.772088 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podStartSLOduration=6.772074062 podStartE2EDuration="6.772074062s" podCreationTimestamp="2026-04-24 17:12:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:12:38.768532494 +0000 UTC m=+1999.517703254" watchObservedRunningTime="2026-04-24 17:12:38.772074062 +0000 UTC m=+1999.521244823" Apr 24 17:12:43.153013 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:43.152971 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:43.153013 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:43.153025 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:12:43.154759 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:43.154725 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:12:43.779893 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:43.779838 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="llm-d-routing-sidecar" containerID="cri-o://f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" gracePeriod=2 Apr 24 17:12:44.072083 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.072058 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6b94f45878-fglj7_d3dc5aca-d101-4c93-95c5-611d1543e22c/main/0.log" Apr 24 17:12:44.072846 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.072823 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:12:44.075845 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.075823 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:12:44.181697 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181633 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181714 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181739 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181761 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65zk6\" (UniqueName: \"kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181793 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181822 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181851 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181927 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181960 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.181921 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182001 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtz7j\" (UniqueName: \"kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182047 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182082 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location\") pod \"7b857573-4789-4ca6-9fe6-660c9283a66d\" (UID: \"7b857573-4789-4ca6-9fe6-660c9283a66d\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182102 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182164 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182125 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location\") pod \"d3dc5aca-d101-4c93-95c5-611d1543e22c\" (UID: \"d3dc5aca-d101-4c93-95c5-611d1543e22c\") " Apr 24 17:12:44.182893 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182373 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home" (OuterVolumeSpecName: "home") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.184066 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182531 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.184066 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.184013 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.184066 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.182810 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.184294 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.184143 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home" (OuterVolumeSpecName: "home") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.185194 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.184642 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm" (OuterVolumeSpecName: "dshm") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.185459 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.185435 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j" (OuterVolumeSpecName: "kube-api-access-qtz7j") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "kube-api-access-qtz7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:12:44.186075 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.186017 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:12:44.186075 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.186044 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:12:44.186375 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.186354 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm" (OuterVolumeSpecName: "dshm") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.186548 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.186528 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6" (OuterVolumeSpecName: "kube-api-access-65zk6") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "kube-api-access-65zk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:12:44.199994 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.199957 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.201426 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.201400 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.248517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.248476 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b857573-4789-4ca6-9fe6-660c9283a66d" (UID: "7b857573-4789-4ca6-9fe6-660c9283a66d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.251694 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.251653 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3dc5aca-d101-4c93-95c5-611d1543e22c" (UID: "d3dc5aca-d101-4c93-95c5-611d1543e22c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:12:44.284660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284629 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtz7j\" (UniqueName: \"kubernetes.io/projected/7b857573-4789-4ca6-9fe6-660c9283a66d-kube-api-access-qtz7j\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284655 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284691 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284706 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284715 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284723 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284732 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zk6\" (UniqueName: \"kubernetes.io/projected/d3dc5aca-d101-4c93-95c5-611d1543e22c-kube-api-access-65zk6\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284740 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284748 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b857573-4789-4ca6-9fe6-660c9283a66d-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284756 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3dc5aca-d101-4c93-95c5-611d1543e22c-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284763 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b857573-4789-4ca6-9fe6-660c9283a66d-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.284849 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.284770 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3dc5aca-d101-4c93-95c5-611d1543e22c-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:12:44.772207 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.772168 2561 generic.go:358] "Generic (PLEG): container finished" podID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerID="6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8" exitCode=137 Apr 24 17:12:44.772494 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.772239 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" Apr 24 17:12:44.772494 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.772248 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerDied","Data":"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8"} Apr 24 17:12:44.772494 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.772282 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb" event={"ID":"7b857573-4789-4ca6-9fe6-660c9283a66d","Type":"ContainerDied","Data":"d96b2377e322f491e2e6285848149b8309ac6452e61e1c1a1b88c224b63dc490"} Apr 24 17:12:44.772494 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.772301 2561 scope.go:117] "RemoveContainer" containerID="6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8" Apr 24 17:12:44.773597 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.773576 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6b94f45878-fglj7_d3dc5aca-d101-4c93-95c5-611d1543e22c/main/0.log" Apr 24 17:12:44.774293 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774269 2561 generic.go:358] "Generic (PLEG): container finished" podID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerID="4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" exitCode=137 Apr 24 17:12:44.774293 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774289 2561 generic.go:358] "Generic (PLEG): container finished" podID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerID="f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" exitCode=0 Apr 24 17:12:44.774473 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774329 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerDied","Data":"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5"} Apr 24 17:12:44.774473 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774352 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerDied","Data":"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87"} Apr 24 17:12:44.774473 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774367 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" event={"ID":"d3dc5aca-d101-4c93-95c5-611d1543e22c","Type":"ContainerDied","Data":"35ace01013b414b40480fe945f080fd45e242a585e56e8d4efef76d722a17ecf"} Apr 24 17:12:44.774473 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.774389 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7" Apr 24 17:12:44.781586 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.781377 2561 scope.go:117] "RemoveContainer" containerID="dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806" Apr 24 17:12:44.797259 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.797237 2561 scope.go:117] "RemoveContainer" containerID="6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8" Apr 24 17:12:44.797616 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:12:44.797567 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8\": container with ID starting with 6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8 not found: ID does not exist" containerID="6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8" Apr 24 17:12:44.797754 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.797620 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8"} err="failed to get container status \"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8\": rpc error: code = NotFound desc = could not find container \"6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8\": container with ID starting with 6548811f34c75ca71feb33d3d52eb2ded13d0f556e470253695547107a3ef1c8 not found: ID does not exist" Apr 24 17:12:44.797754 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.797646 2561 scope.go:117] "RemoveContainer" containerID="dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806" Apr 24 17:12:44.798201 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:12:44.798066 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806\": container with ID starting with dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806 not found: ID does not exist" containerID="dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806" Apr 24 17:12:44.798201 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.798117 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806"} err="failed to get container status \"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806\": rpc error: code = NotFound desc = could not find container \"dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806\": container with ID starting with dc049849219e73778e85e8e941de2d8a5dcacca3b5771457cd4116ed0d601806 not found: ID does not exist" Apr 24 17:12:44.798201 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.798140 2561 scope.go:117] "RemoveContainer" containerID="4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" Apr 24 17:12:44.801933 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.801902 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:12:44.803965 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.803926 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6c46cc7f89-vngxb"] Apr 24 17:12:44.809967 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.809948 2561 scope.go:117] "RemoveContainer" containerID="3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802" Apr 24 17:12:44.817059 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.817037 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:12:44.821100 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.821079 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6b94f45878-fglj7"] Apr 24 17:12:44.821947 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.821916 2561 scope.go:117] "RemoveContainer" containerID="f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" Apr 24 17:12:44.829291 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.829269 2561 scope.go:117] "RemoveContainer" containerID="4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" Apr 24 17:12:44.829576 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:12:44.829557 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5\": container with ID starting with 4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5 not found: ID does not exist" containerID="4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" Apr 24 17:12:44.829660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.829589 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5"} err="failed to get container status \"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5\": rpc error: code = NotFound desc = could not find container \"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5\": container with ID starting with 4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5 not found: ID does not exist" Apr 24 17:12:44.829660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.829615 2561 scope.go:117] "RemoveContainer" containerID="3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802" Apr 24 17:12:44.829955 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:12:44.829938 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802\": container with ID starting with 3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802 not found: ID does not exist" containerID="3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802" Apr 24 17:12:44.830001 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.829969 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802"} err="failed to get container status \"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802\": rpc error: code = NotFound desc = could not find container \"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802\": container with ID starting with 3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802 not found: ID does not exist" Apr 24 17:12:44.830001 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.829986 2561 scope.go:117] "RemoveContainer" containerID="f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" Apr 24 17:12:44.830236 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:12:44.830220 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87\": container with ID starting with f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87 not found: ID does not exist" containerID="f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" Apr 24 17:12:44.830278 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830238 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87"} err="failed to get container status \"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87\": rpc error: code = NotFound desc = could not find container \"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87\": container with ID starting with f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87 not found: ID does not exist" Apr 24 17:12:44.830278 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830259 2561 scope.go:117] "RemoveContainer" containerID="4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5" Apr 24 17:12:44.830471 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830454 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5"} err="failed to get container status \"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5\": rpc error: code = NotFound desc = could not find container \"4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5\": container with ID starting with 4fc4a84a0656559740c205b46f532984029cf5cb5d1970b31f9325e22de59af5 not found: ID does not exist" Apr 24 17:12:44.830532 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830470 2561 scope.go:117] "RemoveContainer" containerID="3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802" Apr 24 17:12:44.830753 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830726 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802"} err="failed to get container status \"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802\": rpc error: code = NotFound desc = could not find container \"3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802\": container with ID starting with 3015c24d66d16cd07a3e90fd3ae7029e8622e15d21cc2468fb633b8f3eaaf802 not found: ID does not exist" Apr 24 17:12:44.830843 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830755 2561 scope.go:117] "RemoveContainer" containerID="f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87" Apr 24 17:12:44.831014 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:44.830998 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87"} err="failed to get container status \"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87\": rpc error: code = NotFound desc = could not find container \"f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87\": container with ID starting with f486552e89481e6b7d34aab580786e18ae39ac02562ce6c8a9b1d41e6002ef87 not found: ID does not exist" Apr 24 17:12:45.850839 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:45.850807 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" path="/var/lib/kubelet/pods/7b857573-4789-4ca6-9fe6-660c9283a66d/volumes" Apr 24 17:12:45.851280 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:45.851238 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" path="/var/lib/kubelet/pods/d3dc5aca-d101-4c93-95c5-611d1543e22c/volumes" Apr 24 17:12:53.154033 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:12:53.153983 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:03.153661 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:03.153619 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:07.937592 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:07.937554 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 17:13:07.938057 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:07.937853 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" podUID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" containerName="storage-initializer" containerID="cri-o://6cb16a0cd2b86819286951d9b48dee8585bbd6636066adc0d342d3e9633eea57" gracePeriod=30 Apr 24 17:13:13.153780 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:13.153731 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:23.153503 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:23.153460 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:28.332185 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332153 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:13:28.332568 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332533 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="llm-d-routing-sidecar" Apr 24 17:13:28.332568 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332545 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="llm-d-routing-sidecar" Apr 24 17:13:28.332568 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332554 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="storage-initializer" Apr 24 17:13:28.332568 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332560 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="storage-initializer" Apr 24 17:13:28.332568 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332569 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="storage-initializer" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332575 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="storage-initializer" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332593 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332598 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332605 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332610 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332662 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="llm-d-routing-sidecar" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332685 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b857573-4789-4ca6-9fe6-660c9283a66d" containerName="main" Apr 24 17:13:28.332752 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.332693 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="d3dc5aca-d101-4c93-95c5-611d1543e22c" containerName="main" Apr 24 17:13:28.337443 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.337426 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.340125 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.340101 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"conv-test-criticality-kserve-self-signed-certs\"" Apr 24 17:13:28.346490 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.346465 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:13:28.400177 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400177 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400274 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7w7\" (UniqueName: \"kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.400432 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.400355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.501786 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.501746 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7w7\" (UniqueName: \"kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.501786 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.501790 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502059 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.501903 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502059 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.501956 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502059 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502045 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502236 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502071 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502236 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502100 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502236 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502155 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502414 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502261 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502414 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502337 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.502588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.502420 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.504402 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.504380 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.504717 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.504693 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.510439 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.510415 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7w7\" (UniqueName: \"kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7\") pod \"conv-test-criticality-kserve-78c74cf566-cscd6\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.648640 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.648591 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:28.775281 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.775252 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:13:28.778801 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:13:28.778769 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5e5c02_a5a6_4e4d_849f_070c8cfcc037.slice/crio-70d324ad6111ec5bfea48f100dc4533a4eda3b44b366baf952715f0143daa3f3 WatchSource:0}: Error finding container 70d324ad6111ec5bfea48f100dc4533a4eda3b44b366baf952715f0143daa3f3: Status 404 returned error can't find the container with id 70d324ad6111ec5bfea48f100dc4533a4eda3b44b366baf952715f0143daa3f3 Apr 24 17:13:28.937944 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.937855 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerStarted","Data":"dadb417a134cedc74dcf202715aa88225aa7e4605ca318f7b0fa61b848756144"} Apr 24 17:13:28.937944 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:28.937894 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerStarted","Data":"70d324ad6111ec5bfea48f100dc4533a4eda3b44b366baf952715f0143daa3f3"} Apr 24 17:13:32.955289 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:32.955253 2561 generic.go:358] "Generic (PLEG): container finished" podID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerID="dadb417a134cedc74dcf202715aa88225aa7e4605ca318f7b0fa61b848756144" exitCode=0 Apr 24 17:13:32.955708 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:32.955343 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerDied","Data":"dadb417a134cedc74dcf202715aa88225aa7e4605ca318f7b0fa61b848756144"} Apr 24 17:13:33.153573 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:33.153530 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:33.961151 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:33.961111 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerStarted","Data":"4f2de36ef7432e01d655a42ddd818589a0f874e102db7e7408c5fc31aa1055e7"} Apr 24 17:13:33.984358 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:33.984297 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" podStartSLOduration=5.98427757 podStartE2EDuration="5.98427757s" podCreationTimestamp="2026-04-24 17:13:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:13:33.981741364 +0000 UTC m=+2054.730912123" watchObservedRunningTime="2026-04-24 17:13:33.98427757 +0000 UTC m=+2054.733448331" Apr 24 17:13:34.417484 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.417447 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48"] Apr 24 17:13:34.421227 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.421202 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.423863 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.423836 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\"" Apr 24 17:13:34.433023 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.432999 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48"] Apr 24 17:13:34.460202 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460163 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tmp-dir\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460202 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460207 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fds\" (UniqueName: \"kubernetes.io/projected/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kube-api-access-h2fds\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460411 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460262 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-home\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460411 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tls-certs\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460411 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460333 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kserve-provision-location\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460411 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460375 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-model-cache\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.460549 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.460435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-dshm\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.561680 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561636 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tls-certs\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.561882 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561687 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kserve-provision-location\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.561882 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561717 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-model-cache\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.561882 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561764 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-dshm\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562060 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561915 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tmp-dir\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562060 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.561977 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fds\" (UniqueName: \"kubernetes.io/projected/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kube-api-access-h2fds\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562060 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.562039 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-home\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562219 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.562181 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kserve-provision-location\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562393 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.562370 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-home\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562512 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.562491 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tmp-dir\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.562573 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.562533 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-model-cache\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.564000 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.563973 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-dshm\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.564493 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.564468 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-tls-certs\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.570839 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.570807 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fds\" (UniqueName: \"kubernetes.io/projected/8ca6d4da-51a7-44f0-bcb1-4d21c228a806-kube-api-access-h2fds\") pod \"stop-feature-test-kserve-866df6c77b-lrc48\" (UID: \"8ca6d4da-51a7-44f0-bcb1-4d21c228a806\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.736238 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.736146 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" Apr 24 17:13:34.872974 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.872942 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48"] Apr 24 17:13:34.874200 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:13:34.874161 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca6d4da_51a7_44f0_bcb1_4d21c228a806.slice/crio-a0cb22cadb52d4027de8d46e515f913d3813b6ad16f51ecd15295cab7db799d1 WatchSource:0}: Error finding container a0cb22cadb52d4027de8d46e515f913d3813b6ad16f51ecd15295cab7db799d1: Status 404 returned error can't find the container with id a0cb22cadb52d4027de8d46e515f913d3813b6ad16f51ecd15295cab7db799d1 Apr 24 17:13:34.968017 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.967320 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" event={"ID":"8ca6d4da-51a7-44f0-bcb1-4d21c228a806","Type":"ContainerStarted","Data":"22bf1babf48f723081c72220f2f09becdb9d43c0e9368ade2b659781cc5f1fcd"} Apr 24 17:13:34.968017 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:34.967369 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-866df6c77b-lrc48" event={"ID":"8ca6d4da-51a7-44f0-bcb1-4d21c228a806","Type":"ContainerStarted","Data":"a0cb22cadb52d4027de8d46e515f913d3813b6ad16f51ecd15295cab7db799d1"} Apr 24 17:13:35.902884 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:35.900262 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:13:35.902884 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:35.901014 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="main" containerID="cri-o://4f2de36ef7432e01d655a42ddd818589a0f874e102db7e7408c5fc31aa1055e7" gracePeriod=30 Apr 24 17:13:37.981260 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:37.981229 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-646bd5b44b-sb2f6_20bc3cbf-4956-4618-82dd-d1e5d8fb106d/storage-initializer/0.log" Apr 24 17:13:37.981627 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:37.981272 2561 generic.go:358] "Generic (PLEG): container finished" podID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" containerID="6cb16a0cd2b86819286951d9b48dee8585bbd6636066adc0d342d3e9633eea57" exitCode=137 Apr 24 17:13:37.981627 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:37.981307 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" event={"ID":"20bc3cbf-4956-4618-82dd-d1e5d8fb106d","Type":"ContainerDied","Data":"6cb16a0cd2b86819286951d9b48dee8585bbd6636066adc0d342d3e9633eea57"} Apr 24 17:13:38.619505 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.619481 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-646bd5b44b-sb2f6_20bc3cbf-4956-4618-82dd-d1e5d8fb106d/storage-initializer/0.log" Apr 24 17:13:38.619654 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.619555 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 17:13:38.649645 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.649611 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:13:38.708344 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708249 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708344 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708297 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708350 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnktv\" (UniqueName: \"kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708379 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708572 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708546 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708771 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708617 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708771 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708713 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs\") pod \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\" (UID: \"20bc3cbf-4956-4618-82dd-d1e5d8fb106d\") " Apr 24 17:13:38.708875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708553 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home" (OuterVolumeSpecName: "home") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:13:38.708875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708707 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache" (OuterVolumeSpecName: "model-cache") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:13:38.708875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.708849 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:13:38.709146 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.709117 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.709146 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.709146 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.709272 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.709159 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.711202 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.711173 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv" (OuterVolumeSpecName: "kube-api-access-nnktv") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "kube-api-access-nnktv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:13:38.711360 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.711323 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:13:38.711561 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.711539 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm" (OuterVolumeSpecName: "dshm") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:13:38.725487 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.725455 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20bc3cbf-4956-4618-82dd-d1e5d8fb106d" (UID: "20bc3cbf-4956-4618-82dd-d1e5d8fb106d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:13:38.810455 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.810417 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnktv\" (UniqueName: \"kubernetes.io/projected/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kube-api-access-nnktv\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.810455 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.810459 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.810625 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.810476 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.810625 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.810493 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20bc3cbf-4956-4618-82dd-d1e5d8fb106d-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:13:38.986524 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.986443 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-646bd5b44b-sb2f6_20bc3cbf-4956-4618-82dd-d1e5d8fb106d/storage-initializer/0.log" Apr 24 17:13:38.986946 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.986551 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" event={"ID":"20bc3cbf-4956-4618-82dd-d1e5d8fb106d","Type":"ContainerDied","Data":"46f5863be45cbbb07a8466432f496b511bbbbec2d9dede6566ffc0c7813b4a5a"} Apr 24 17:13:38.986946 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.986563 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6" Apr 24 17:13:38.986946 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:38.986594 2561 scope.go:117] "RemoveContainer" containerID="6cb16a0cd2b86819286951d9b48dee8585bbd6636066adc0d342d3e9633eea57" Apr 24 17:13:39.026199 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:39.026167 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 17:13:39.032799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:39.032768 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-646bd5b44b-sb2f6"] Apr 24 17:13:39.850744 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:39.850714 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" path="/var/lib/kubelet/pods/20bc3cbf-4956-4618-82dd-d1e5d8fb106d/volumes" Apr 24 17:13:43.153744 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:43.153705 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:13:53.153631 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:13:53.153579 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:14:03.153502 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:03.153452 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused" Apr 24 17:14:06.084751 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.084722 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-criticality-kserve-78c74cf566-cscd6_bf5e5c02-a5a6-4e4d-849f-070c8cfcc037/main/0.log" Apr 24 17:14:06.085162 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.085136 2561 generic.go:358] "Generic (PLEG): container finished" podID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerID="4f2de36ef7432e01d655a42ddd818589a0f874e102db7e7408c5fc31aa1055e7" exitCode=137 Apr 24 17:14:06.085246 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.085216 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerDied","Data":"4f2de36ef7432e01d655a42ddd818589a0f874e102db7e7408c5fc31aa1055e7"} Apr 24 17:14:06.101091 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.101072 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-criticality-kserve-78c74cf566-cscd6_bf5e5c02-a5a6-4e4d-849f-070c8cfcc037/main/0.log" Apr 24 17:14:06.101542 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.101518 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:14:06.192309 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192230 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192309 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192262 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192309 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192290 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192313 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192342 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192386 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192450 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw7w7\" (UniqueName: \"kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7\") pod \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\" (UID: \"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037\") " Apr 24 17:14:06.192815 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192591 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home" (OuterVolumeSpecName: "home") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:06.192815 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192690 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:06.192917 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192844 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:06.192957 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192926 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.192957 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192947 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.193017 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.192961 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.194698 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.194652 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:14:06.194825 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.194703 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7" (OuterVolumeSpecName: "kube-api-access-pw7w7") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "kube-api-access-pw7w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:14:06.194825 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.194750 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm" (OuterVolumeSpecName: "dshm") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:06.263838 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.263792 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" (UID: "bf5e5c02-a5a6-4e4d-849f-070c8cfcc037"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:06.294070 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.294034 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw7w7\" (UniqueName: \"kubernetes.io/projected/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kube-api-access-pw7w7\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.294070 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.294070 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.294281 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.294084 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:06.294281 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:06.294097 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:07.089853 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.089820 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-criticality-kserve-78c74cf566-cscd6_bf5e5c02-a5a6-4e4d-849f-070c8cfcc037/main/0.log" Apr 24 17:14:07.090341 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.090318 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" Apr 24 17:14:07.090423 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.090312 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6" event={"ID":"bf5e5c02-a5a6-4e4d-849f-070c8cfcc037","Type":"ContainerDied","Data":"70d324ad6111ec5bfea48f100dc4533a4eda3b44b366baf952715f0143daa3f3"} Apr 24 17:14:07.090481 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.090447 2561 scope.go:117] "RemoveContainer" containerID="4f2de36ef7432e01d655a42ddd818589a0f874e102db7e7408c5fc31aa1055e7" Apr 24 17:14:07.102766 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.102736 2561 scope.go:117] "RemoveContainer" containerID="dadb417a134cedc74dcf202715aa88225aa7e4605ca318f7b0fa61b848756144" Apr 24 17:14:07.127774 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.127726 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:14:07.130152 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.130131 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/conv-test-criticality-kserve-78c74cf566-cscd6"] Apr 24 17:14:07.850158 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:07.850119 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" path="/var/lib/kubelet/pods/bf5e5c02-a5a6-4e4d-849f-070c8cfcc037/volumes" Apr 24 17:14:13.163126 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:13.163097 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:14:13.171259 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:13.171234 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:14:17.438101 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:17.438072 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:14:17.438558 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:17.438328 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" containerID="cri-o://3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7" gracePeriod=30 Apr 24 17:14:19.924748 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:19.924720 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:14:19.928446 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:19.928426 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:14:31.346032 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346000 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:14:31.346517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346493 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" containerName="storage-initializer" Apr 24 17:14:31.346517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346517 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" containerName="storage-initializer" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346533 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="storage-initializer" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346542 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="storage-initializer" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346553 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="main" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346558 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="main" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346641 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="bf5e5c02-a5a6-4e4d-849f-070c8cfcc037" containerName="main" Apr 24 17:14:31.346730 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.346658 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="20bc3cbf-4956-4618-82dd-d1e5d8fb106d" containerName="storage-initializer" Apr 24 17:14:31.350098 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.350078 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.352547 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.352524 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-97qfp\"" Apr 24 17:14:31.352654 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.352568 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\"" Apr 24 17:14:31.361580 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.361554 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:14:31.526598 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526804 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526804 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526660 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526804 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526776 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526920 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526920 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.526920 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.526877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm589\" (UniqueName: \"kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627648 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627618 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627659 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627749 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627778 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627823 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.627852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627847 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gm589\" (UniqueName: \"kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.628115 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.627930 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.628115 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.628046 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.628217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.628140 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.628217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.628173 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.628217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.628197 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.629994 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.629971 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.630169 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.630152 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.635836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.635809 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm589\" (UniqueName: \"kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.660910 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.660882 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:31.793330 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.793292 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:14:31.796580 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:14:31.796550 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865e0dfb_de16_402a_9e42_618d9b4a7bab.slice/crio-20ea42ece001af989d8c925c03e7d741794f5f14ade3d3cf629b1ea15babbe3e WatchSource:0}: Error finding container 20ea42ece001af989d8c925c03e7d741794f5f14ade3d3cf629b1ea15babbe3e: Status 404 returned error can't find the container with id 20ea42ece001af989d8c925c03e7d741794f5f14ade3d3cf629b1ea15babbe3e Apr 24 17:14:31.798772 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:31.798752 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 17:14:32.176660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:32.176627 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerStarted","Data":"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825"} Apr 24 17:14:32.176660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:32.176662 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerStarted","Data":"20ea42ece001af989d8c925c03e7d741794f5f14ade3d3cf629b1ea15babbe3e"} Apr 24 17:14:37.199517 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:37.199482 2561 generic.go:358] "Generic (PLEG): container finished" podID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerID="0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825" exitCode=0 Apr 24 17:14:37.200072 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:37.199562 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerDied","Data":"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825"} Apr 24 17:14:38.205112 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:38.205061 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerStarted","Data":"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781"} Apr 24 17:14:38.226872 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:38.226809 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=7.226790357 podStartE2EDuration="7.226790357s" podCreationTimestamp="2026-04-24 17:14:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:14:38.22571672 +0000 UTC m=+2118.974887505" watchObservedRunningTime="2026-04-24 17:14:38.226790357 +0000 UTC m=+2118.975961120" Apr 24 17:14:41.661563 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:41.661533 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:14:41.663224 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:41.663191 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:14:47.709289 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.709265 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh_9251a216-8fc7-4bda-8bd0-a70dacd03003/main/0.log" Apr 24 17:14:47.709640 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.709618 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:14:47.788467 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788437 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgcng\" (UniqueName: \"kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788467 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788479 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788760 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788507 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788760 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788553 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788760 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788633 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788760 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788727 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.788957 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.788766 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm\") pod \"9251a216-8fc7-4bda-8bd0-a70dacd03003\" (UID: \"9251a216-8fc7-4bda-8bd0-a70dacd03003\") " Apr 24 17:14:47.789141 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.789100 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache" (OuterVolumeSpecName: "model-cache") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:47.789350 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.789314 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home" (OuterVolumeSpecName: "home") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:47.790963 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.790921 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm" (OuterVolumeSpecName: "dshm") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:47.791262 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.791237 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng" (OuterVolumeSpecName: "kube-api-access-zgcng") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "kube-api-access-zgcng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:14:47.791331 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.791280 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:14:47.800898 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.800868 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:47.841848 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.841807 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9251a216-8fc7-4bda-8bd0-a70dacd03003" (UID: "9251a216-8fc7-4bda-8bd0-a70dacd03003"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:14:47.890610 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890578 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890610 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890611 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890624 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890637 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890653 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgcng\" (UniqueName: \"kubernetes.io/projected/9251a216-8fc7-4bda-8bd0-a70dacd03003-kube-api-access-zgcng\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890690 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9251a216-8fc7-4bda-8bd0-a70dacd03003-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:47.890836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:47.890705 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9251a216-8fc7-4bda-8bd0-a70dacd03003-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:14:48.242000 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.241970 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh_9251a216-8fc7-4bda-8bd0-a70dacd03003/main/0.log" Apr 24 17:14:48.242359 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.242335 2561 generic.go:358] "Generic (PLEG): container finished" podID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerID="3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7" exitCode=137 Apr 24 17:14:48.242427 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.242375 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerDied","Data":"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7"} Apr 24 17:14:48.242427 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.242405 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" Apr 24 17:14:48.242427 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.242420 2561 scope.go:117] "RemoveContainer" containerID="3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7" Apr 24 17:14:48.242556 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.242407 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh" event={"ID":"9251a216-8fc7-4bda-8bd0-a70dacd03003","Type":"ContainerDied","Data":"c3cc2d91b9a0fef2d9ac7b742e29f6628ecbecb9705dc8e9c911d8a3875a0753"} Apr 24 17:14:48.250814 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.250796 2561 scope.go:117] "RemoveContainer" containerID="94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405" Apr 24 17:14:48.262039 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.262013 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:14:48.265456 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.265430 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-686d8cc94828xdh"] Apr 24 17:14:48.310892 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.310865 2561 scope.go:117] "RemoveContainer" containerID="3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7" Apr 24 17:14:48.311238 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:14:48.311217 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7\": container with ID starting with 3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7 not found: ID does not exist" containerID="3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7" Apr 24 17:14:48.311323 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.311252 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7"} err="failed to get container status \"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7\": rpc error: code = NotFound desc = could not find container \"3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7\": container with ID starting with 3d692b3b8f9c84c6442395488cf00f0fea5737c45b5b9184c8b6d408f8affda7 not found: ID does not exist" Apr 24 17:14:48.311323 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.311277 2561 scope.go:117] "RemoveContainer" containerID="94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405" Apr 24 17:14:48.311600 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:14:48.311581 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405\": container with ID starting with 94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405 not found: ID does not exist" containerID="94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405" Apr 24 17:14:48.311639 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:48.311607 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405"} err="failed to get container status \"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405\": rpc error: code = NotFound desc = could not find container \"94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405\": container with ID starting with 94bf31d6840f3791bcdb2034149a7ac114ad66e4d1ddca64e6cf0b51d8553405 not found: ID does not exist" Apr 24 17:14:49.850367 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:49.850334 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" path="/var/lib/kubelet/pods/9251a216-8fc7-4bda-8bd0-a70dacd03003/volumes" Apr 24 17:14:51.662087 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:14:51.662050 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:01.661214 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:01.661173 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:15:01.661759 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:01.661494 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:11.662012 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:11.661968 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:21.661546 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:21.661501 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:31.661765 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:31.661723 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:41.662133 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:41.662084 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:15:51.661603 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:15:51.661555 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:16:01.661653 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:01.661611 2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused" Apr 24 17:16:11.671458 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:11.671419 2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:16:11.678553 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:11.678523 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:16:19.287347 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:19.287313 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:16:19.287839 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:19.287581 2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" containerID="cri-o://7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781" gracePeriod=30 Apr 24 17:16:20.027555 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.027533 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:16:20.172240 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172213 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172406 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172250 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172406 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172282 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172406 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172298 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm589\" (UniqueName: \"kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172406 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172318 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172406 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172405 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172719 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172427 2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs\") pod \"865e0dfb-de16-402a-9e42-618d9b4a7bab\" (UID: \"865e0dfb-de16-402a-9e42-618d9b4a7bab\") " Apr 24 17:16:20.172719 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172520 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache" (OuterVolumeSpecName: "model-cache") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:16:20.172837 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172808 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.173032 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.172996 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home" (OuterVolumeSpecName: "home") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:16:20.174496 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.174464 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589" (OuterVolumeSpecName: "kube-api-access-gm589") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "kube-api-access-gm589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 24 17:16:20.174611 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.174585 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 24 17:16:20.174845 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.174822 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm" (OuterVolumeSpecName: "dshm") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:16:20.184712 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.184687 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:16:20.231559 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.231528 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "865e0dfb-de16-402a-9e42-618d9b4a7bab" (UID: "865e0dfb-de16-402a-9e42-618d9b4a7bab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 24 17:16:20.273874 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273853 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.273874 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273873 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865e0dfb-de16-402a-9e42-618d9b4a7bab-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.274011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273882 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.274011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273891 2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-kserve-provision-location\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.274011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273900 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm589\" (UniqueName: \"kubernetes.io/projected/865e0dfb-de16-402a-9e42-618d9b4a7bab-kube-api-access-gm589\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.274011 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.273908 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865e0dfb-de16-402a-9e42-618d9b4a7bab-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\"" Apr 24 17:16:20.566725 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.566624 2561 generic.go:358] "Generic (PLEG): container finished" podID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerID="7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781" exitCode=0 Apr 24 17:16:20.566725 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.566716 2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" Apr 24 17:16:20.567148 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.566710 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerDied","Data":"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781"} Apr 24 17:16:20.567148 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.566821 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"865e0dfb-de16-402a-9e42-618d9b4a7bab","Type":"ContainerDied","Data":"20ea42ece001af989d8c925c03e7d741794f5f14ade3d3cf629b1ea15babbe3e"} Apr 24 17:16:20.567148 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.566837 2561 scope.go:117] "RemoveContainer" containerID="7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781" Apr 24 17:16:20.575353 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.575340 2561 scope.go:117] "RemoveContainer" containerID="0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825" Apr 24 17:16:20.588132 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.588111 2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:16:20.592063 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.592031 2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"] Apr 24 17:16:20.592350 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.592330 2561 scope.go:117] "RemoveContainer" containerID="7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781" Apr 24 17:16:20.592569 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:16:20.592554 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781\": container with ID starting with 7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781 not found: ID does not exist" containerID="7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781" Apr 24 17:16:20.592632 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.592576 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781"} err="failed to get container status \"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781\": rpc error: code = NotFound desc = could not find container \"7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781\": container with ID starting with 7a8a69ff90c369445e3d41060e969fe803b976604b102fa527ee5ed13cb96781 not found: ID does not exist" Apr 24 17:16:20.592632 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.592592 2561 scope.go:117] "RemoveContainer" containerID="0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825" Apr 24 17:16:20.592818 ip-10-0-139-51 kubenswrapper[2561]: E0424 17:16:20.592800 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825\": container with ID starting with 0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825 not found: ID does not exist" containerID="0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825" Apr 24 17:16:20.592861 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:20.592824 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825"} err="failed to get container status \"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825\": rpc error: code = NotFound desc = could not find container \"0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825\": container with ID starting with 0112b4449aa4a006edcca7ceea56fc66ba80261705e64cd03b7396d6d735e825 not found: ID does not exist" Apr 24 17:16:21.850062 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:16:21.850030 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" path="/var/lib/kubelet/pods/865e0dfb-de16-402a-9e42-618d9b4a7bab/volumes" Apr 24 17:19:19.946370 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:19:19.946336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:19:19.951080 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:19:19.951050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:24:19.970205 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:24:19.970122 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:24:19.978118 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:24:19.978097 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:29:19.996602 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:19.996497 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:29:20.001628 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:20.001608 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:29:34.261584 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:34.261556 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:34.340830 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:34.340796 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:35.332167 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:35.332140 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:35.378126 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:35.378098 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:36.361358 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:36.361327 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:36.404027 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:36.403991 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:37.360563 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:37.360535 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:37.400711 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:37.400658 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:38.372522 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:38.372490 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:38.411181 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:38.411155 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:39.368513 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:39.368485 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:39.406222 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:39.406196 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:40.361848 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:40.361822 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:40.402102 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:40.402078 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:41.365716 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:41.365681 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:41.405033 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:41.405005 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:42.369836 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:42.369806 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:42.409040 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:42.409013 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:43.379600 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:43.379569 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:43.418567 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:43.418538 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:44.397834 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:44.397796 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:44.439245 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:44.439216 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:45.423729 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:45.423701 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:45.464330 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:45.464299 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:46.447654 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:46.447624 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:46.490143 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:46.490113 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:47.500720 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:47.500682 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-zd2ql_43233bc0-0078-446d-b670-b8c8159a4a95/istio-proxy/0.log" Apr 24 17:29:47.545624 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:47.545595 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-866df6c77b-lrc48_8ca6d4da-51a7-44f0-bcb1-4d21c228a806/storage-initializer/0.log" Apr 24 17:29:48.591745 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:48.591711 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-5lfr4_c144f2dd-e507-4c01-98b6-489818cb43f9/istio-proxy/0.log" Apr 24 17:29:48.610188 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:48.610163 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6f5d86d4-87vd7_15b66498-b5b4-403e-8f0c-6754367fa215/router/0.log" Apr 24 17:29:49.385277 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:49.385249 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-5lfr4_c144f2dd-e507-4c01-98b6-489818cb43f9/istio-proxy/0.log" Apr 24 17:29:49.402480 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:49.402447 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6f5d86d4-87vd7_15b66498-b5b4-403e-8f0c-6754367fa215/router/0.log" Apr 24 17:29:50.207211 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:50.207172 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-5lfr4_c144f2dd-e507-4c01-98b6-489818cb43f9/istio-proxy/0.log" Apr 24 17:29:50.222953 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:50.222931 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6f5d86d4-87vd7_15b66498-b5b4-403e-8f0c-6754367fa215/router/0.log" Apr 24 17:29:50.965038 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:50.965006 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:29:50.994064 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:50.994032 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:29:51.081860 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:51.081835 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:29:51.876256 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:51.876227 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:29:51.901748 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:51.901728 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:29:51.974217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:51.974190 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:29:52.761081 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:52.761052 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:29:52.788699 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:52.788657 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:29:52.860936 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:52.860907 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:29:53.641522 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:53.641495 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:29:53.666550 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:53.666523 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:29:53.735754 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:53.735727 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:29:54.515747 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:54.515715 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:29:54.542272 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:54.542246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:29:54.610945 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:29:54.610916 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:30:00.132534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:00.132504 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dx77w_a768d2bb-7b4b-4040-94aa-1bf6370247c1/global-pull-secret-syncer/0.log" Apr 24 17:30:00.205042 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:00.205009 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-srzmm_813b1e82-26e4-4a64-a2d6-d4bc774b92e2/konnectivity-agent/0.log" Apr 24 17:30:00.306070 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:00.306044 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-51.ec2.internal_964185a562d5ec022f05355b7dc81013/haproxy/0.log" Apr 24 17:30:04.630588 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:04.630561 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-lv7qb_66b93258-4aa0-400c-b3c5-01332849494b/authorino/0.log" Apr 24 17:30:04.700901 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:04.700879 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-9ht4r_0db6e30d-4e12-4301-af19-38fb78346d82/manager/0.log" Apr 24 17:30:04.850421 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:04.850388 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-ftb2d_c14c9346-0f21-4225-a096-22fe8163a745/manager/0.log" Apr 24 17:30:05.847860 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.847830 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/alertmanager/0.log" Apr 24 17:30:05.873046 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.873024 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/config-reloader/0.log" Apr 24 17:30:05.895293 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.895269 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/kube-rbac-proxy-web/0.log" Apr 24 17:30:05.917051 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.917027 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/kube-rbac-proxy/0.log" Apr 24 17:30:05.940816 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.940769 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/kube-rbac-proxy-metric/0.log" Apr 24 17:30:05.962617 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.962595 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/prom-label-proxy/0.log" Apr 24 17:30:05.992706 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:05.992661 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_907fcbea-0f80-44fd-8371-f1a7f88e328f/init-config-reloader/0.log" Apr 24 17:30:06.064687 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.064640 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-state-metrics/0.log" Apr 24 17:30:06.093789 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.093769 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-rbac-proxy-main/0.log" Apr 24 17:30:06.119833 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.119812 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-r4f2j_10734571-7a9a-4ec7-b1bd-53f3f501d9f3/kube-rbac-proxy-self/0.log" Apr 24 17:30:06.297217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.297187 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/node-exporter/0.log" Apr 24 17:30:06.318389 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.318366 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/kube-rbac-proxy/0.log" Apr 24 17:30:06.343341 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.343277 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4r9fk_47e1def0-fc20-457c-b76c-01fba0d99c84/init-textfile/0.log" Apr 24 17:30:06.560966 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.560940 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/prometheus/0.log" Apr 24 17:30:06.582596 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.582575 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/config-reloader/0.log" Apr 24 17:30:06.606658 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.606594 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/thanos-sidecar/0.log" Apr 24 17:30:06.633063 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.633032 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/kube-rbac-proxy-web/0.log" Apr 24 17:30:06.654814 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.654792 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/kube-rbac-proxy/0.log" Apr 24 17:30:06.681611 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.681590 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/kube-rbac-proxy-thanos/0.log" Apr 24 17:30:06.708304 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.708287 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_18fd18e7-cf1c-46bf-99b6-518f009ede0e/init-config-reloader/0.log" Apr 24 17:30:06.786899 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.786874 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-9kc99_7cfc197c-9833-4185-8321-b5df76b2eea1/prometheus-operator-admission-webhook/0.log" Apr 24 17:30:06.815910 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.815889 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-857678c95-pmn4r_b09e332a-bc47-445b-ba0f-8b6740d2419b/telemeter-client/0.log" Apr 24 17:30:06.839689 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.839656 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-857678c95-pmn4r_b09e332a-bc47-445b-ba0f-8b6740d2419b/reload/0.log" Apr 24 17:30:06.862715 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:06.862690 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-857678c95-pmn4r_b09e332a-bc47-445b-ba0f-8b6740d2419b/kube-rbac-proxy/0.log" Apr 24 17:30:08.127829 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:08.127803 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-cb95c66f6-6c8j4_142e9821-410f-44b3-9366-6e7f6419ac79/networking-console-plugin/0.log" Apr 24 17:30:09.375516 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375478 2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r"] Apr 24 17:30:09.375921 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375896 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" Apr 24 17:30:09.375921 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375909 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" Apr 24 17:30:09.375921 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375920 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375926 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375940 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="storage-initializer" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375948 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="storage-initializer" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375964 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="storage-initializer" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.375970 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="storage-initializer" Apr 24 17:30:09.376024 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.376024 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="9251a216-8fc7-4bda-8bd0-a70dacd03003" containerName="main" Apr 24 17:30:09.376200 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.376034 2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="865e0dfb-de16-402a-9e42-618d9b4a7bab" containerName="main" Apr 24 17:30:09.379134 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.379114 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.381514 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.381492 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mg2\"/\"openshift-service-ca.crt\"" Apr 24 17:30:09.381606 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.381492 2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mg2\"/\"kube-root-ca.crt\"" Apr 24 17:30:09.382600 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.382582 2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47mg2\"/\"default-dockercfg-bn9xv\"" Apr 24 17:30:09.388841 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.388819 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r"] Apr 24 17:30:09.551603 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.551561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xfb\" (UniqueName: \"kubernetes.io/projected/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-kube-api-access-c7xfb\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.551799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.551622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-sys\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.551799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.551735 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-lib-modules\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.551799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.551757 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-proc\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.551799 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.551771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-podres\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652324 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652233 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-lib-modules\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652324 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652277 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-proc\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652324 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652299 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-podres\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652379 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xfb\" (UniqueName: \"kubernetes.io/projected/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-kube-api-access-c7xfb\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652384 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-proc\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652429 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-lib-modules\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652442 2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-sys\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652534 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652476 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-podres\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.652723 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.652554 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-sys\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.660613 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.660592 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xfb\" (UniqueName: \"kubernetes.io/projected/0399bb9d-30af-4b13-8938-9bfc1eb42ca6-kube-api-access-c7xfb\") pod \"perf-node-gather-daemonset-8tk8r\" (UID: \"0399bb9d-30af-4b13-8938-9bfc1eb42ca6\") " pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:09.689660 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:09.689624 2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:10.014659 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.014632 2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r"] Apr 24 17:30:10.016604 ip-10-0-139-51 kubenswrapper[2561]: W0424 17:30:10.016569 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0399bb9d_30af_4b13_8938_9bfc1eb42ca6.slice/crio-1e32c4d032c14f9909a5b1de1dd5c51675259dc4693cdd40c17b1b730c7dea6f WatchSource:0}: Error finding container 1e32c4d032c14f9909a5b1de1dd5c51675259dc4693cdd40c17b1b730c7dea6f: Status 404 returned error can't find the container with id 1e32c4d032c14f9909a5b1de1dd5c51675259dc4693cdd40c17b1b730c7dea6f Apr 24 17:30:10.018142 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.018121 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 24 17:30:10.314875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.314797 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" event={"ID":"0399bb9d-30af-4b13-8938-9bfc1eb42ca6","Type":"ContainerStarted","Data":"e924e88b4ce0ee57bd00d4efe19b4c29aeadeec715a358d92e0750b5c6ecc840"} Apr 24 17:30:10.314875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.314827 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqv7x_9d591d2a-71f2-4e1f-9e37-a4a1756a08f3/dns/0.log" Apr 24 17:30:10.314875 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.314839 2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" event={"ID":"0399bb9d-30af-4b13-8938-9bfc1eb42ca6","Type":"ContainerStarted","Data":"1e32c4d032c14f9909a5b1de1dd5c51675259dc4693cdd40c17b1b730c7dea6f"} Apr 24 17:30:10.315082 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.315034 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:10.331029 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.330980 2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" podStartSLOduration=1.33096816 podStartE2EDuration="1.33096816s" podCreationTimestamp="2026-04-24 17:30:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 17:30:10.329766092 +0000 UTC m=+3051.078936852" watchObservedRunningTime="2026-04-24 17:30:10.33096816 +0000 UTC m=+3051.080139116" Apr 24 17:30:10.336266 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.336241 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jqv7x_9d591d2a-71f2-4e1f-9e37-a4a1756a08f3/kube-rbac-proxy/0.log" Apr 24 17:30:10.502182 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.502154 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrlj6_afd8262e-c6d9-4d7a-89c6-4b87b7281a67/dns-node-resolver/0.log" Apr 24 17:30:10.973435 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.973376 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8b56f5b5b-jclzv_c35319be-d3a9-48a8-9aa6-3e1d55bccb11/registry/0.log" Apr 24 17:30:10.996646 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:10.996619 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4586_2d1c3528-cf11-4ef1-bb88-2f59df7e45a9/node-ca/0.log" Apr 24 17:30:11.868924 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:11.868897 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-5lfr4_c144f2dd-e507-4c01-98b6-489818cb43f9/istio-proxy/0.log" Apr 24 17:30:11.894141 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:11.894118 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6f5d86d4-87vd7_15b66498-b5b4-403e-8f0c-6754367fa215/router/0.log" Apr 24 17:30:12.323003 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:12.322927 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h7m28_780cdd19-26a2-4a6f-844d-748c9e024c5f/serve-healthcheck-canary/0.log" Apr 24 17:30:12.948702 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:12.948659 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xlfzt_9d427b7d-8313-4e70-8dd1-f8ae2049538f/kube-rbac-proxy/0.log" Apr 24 17:30:12.969217 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:12.969190 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xlfzt_9d427b7d-8313-4e70-8dd1-f8ae2049538f/exporter/0.log" Apr 24 17:30:12.989941 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:12.989919 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xlfzt_9d427b7d-8313-4e70-8dd1-f8ae2049538f/extractor/0.log" Apr 24 17:30:15.592161 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:15.592132 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75b5bf9f6d-t87zm_95eb5291-941d-4566-bba9-e5bbbcd96c88/manager/0.log" Apr 24 17:30:16.328244 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:16.328165 2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47mg2/perf-node-gather-daemonset-8tk8r" Apr 24 17:30:16.346510 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:16.346484 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-h5kmf_854369b7-40d4-4e3a-9def-ccd9c921065e/manager/0.log" Apr 24 17:30:22.874891 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.874862 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/kube-multus-additional-cni-plugins/0.log" Apr 24 17:30:22.897042 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.897020 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/egress-router-binary-copy/0.log" Apr 24 17:30:22.916858 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.916839 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/cni-plugins/0.log" Apr 24 17:30:22.936763 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.936748 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/bond-cni-plugin/0.log" Apr 24 17:30:22.958104 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.958085 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/routeoverride-cni/0.log" Apr 24 17:30:22.991075 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:22.991048 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/whereabouts-cni-bincopy/0.log" Apr 24 17:30:23.011915 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:23.011897 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkvcg_b21bb7b3-2bb8-4576-b27b-a786cd88c140/whereabouts-cni/0.log" Apr 24 17:30:23.072617 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:23.072595 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q6scl_da21c390-2975-4210-a740-ee0091b9c5b0/kube-multus/0.log" Apr 24 17:30:23.207244 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:23.207178 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwzw9_b71bdf81-e127-49a6-aca6-ff4846028fd9/network-metrics-daemon/0.log" Apr 24 17:30:23.226489 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:23.226467 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwzw9_b71bdf81-e127-49a6-aca6-ff4846028fd9/kube-rbac-proxy/0.log" Apr 24 17:30:24.671923 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.671892 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-controller/0.log" Apr 24 17:30:24.688852 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.688823 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/0.log" Apr 24 17:30:24.702269 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.702246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovn-acl-logging/1.log" Apr 24 17:30:24.721227 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.721205 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/kube-rbac-proxy-node/0.log" Apr 24 17:30:24.740969 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.740946 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/kube-rbac-proxy-ovn-metrics/0.log" Apr 24 17:30:24.760006 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.759980 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/northd/0.log" Apr 24 17:30:24.779927 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.779907 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/nbdb/0.log" Apr 24 17:30:24.803400 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.803381 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/sbdb/0.log" Apr 24 17:30:24.900521 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:24.900487 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxfbv_dc972117-5b3e-4446-8204-6290e86329ad/ovnkube-controller/0.log" Apr 24 17:30:26.007280 ip-10-0-139-51 kubenswrapper[2561]: I0424 17:30:26.007258 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mhwg5_8f64b904-5057-4953-8d22-8b20d43e4fcf/network-check-target-container/0.log"