Sep 09 19:55:12.288837 ip-10-0-141-196 systemd[1]: Starting Kubernetes Kubelet... Sep 09 19:55:12.691697 ip-10-0-141-196 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 19:55:12.691697 ip-10-0-141-196 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 19:55:12.691697 ip-10-0-141-196 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 19:55:12.691697 ip-10-0-141-196 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 19:55:12.691697 ip-10-0-141-196 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 19:55:12.692692 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.692593 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 19:55:12.697853 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697815 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.697853 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697850 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.697853 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697856 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.697853 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697861 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697874 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697878 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697883 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697887 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697891 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697896 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697902 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697919 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697924 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697960 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697977 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697983 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697989 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.698076 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.697995 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698018 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698171 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698179 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698184 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698188 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698193 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698198 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698202 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698206 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698210 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698214 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698218 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698222 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698226 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698230 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698234 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698238 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698242 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.698447 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698246 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698250 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698257 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698261 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698265 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698269 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698273 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698277 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698283 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698288 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698293 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698298 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698302 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698305 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698309 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698315 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698320 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698324 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698328 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698332 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.698995 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698336 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698340 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698344 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698347 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698349 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698352 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698355 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698358 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698360 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698363 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698365 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698368 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698371 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698373 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698376 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698378 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698381 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698383 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698386 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698388 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.699526 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698391 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698393 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698396 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698398 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698401 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698403 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698406 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698410 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698413 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698416 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698418 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698421 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698424 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698428 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698431 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698433 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698435 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698438 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698440 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698443 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.700045 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698445 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698448 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.698452 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699248 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699257 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699262 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699266 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699269 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699272 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699275 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699277 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699280 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699283 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699285 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699288 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699290 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699293 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699295 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699298 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699301 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.700594 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699305 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699307 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699310 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699312 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699315 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699317 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699320 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699322 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699324 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699327 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699330 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699332 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699335 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699337 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699340 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699342 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699344 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699347 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699349 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699352 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.701131 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699354 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699356 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699360 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699362 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699365 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699367 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699369 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699372 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699375 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699378 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699380 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699384 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699386 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699389 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699392 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699394 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699397 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699399 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699402 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699404 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.701790 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699406 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699410 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699413 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699416 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699418 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699421 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699423 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699426 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699428 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699431 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699433 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699436 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699438 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699440 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699443 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699445 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699447 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699450 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699452 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.702359 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699455 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699457 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699460 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699483 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699488 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699490 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699493 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699497 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699499 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699502 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699504 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699508 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699512 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699515 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699518 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699520 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699523 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699526 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699528 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699531 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.702886 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699534 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699536 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699538 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699944 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699948 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699951 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699953 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699956 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699959 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699961 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699964 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699967 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699969 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699972 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699974 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699977 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699984 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699988 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699991 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.703407 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699995 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.699997 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700000 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700003 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700005 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700008 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700010 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700012 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700015 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700018 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700020 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700023 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700025 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700028 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700030 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700032 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700035 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700037 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700040 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700043 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.703940 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700045 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700048 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700050 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700053 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700056 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700058 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700061 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700063 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700066 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700069 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700072 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700075 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700077 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700081 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700084 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700087 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700089 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700092 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700094 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700097 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.704475 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700100 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700102 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700105 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700107 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700109 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700112 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700114 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700117 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700121 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700124 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700127 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700129 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700132 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700135 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700138 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700141 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700145 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700148 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700150 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.705004 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700153 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700156 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700159 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700161 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700164 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700167 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700170 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700172 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700175 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700177 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700180 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700182 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700184 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700187 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700189 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700191 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700194 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700197 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700199 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700201 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.705525 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700204 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700206 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700208 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.700211 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700305 2573 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700353 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700360 2573 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700365 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700369 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700373 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700379 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700383 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700386 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700390 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700393 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700397 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700400 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700403 2573 flags.go:64] FLAG: --cgroup-root="" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700407 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700410 2573 flags.go:64] FLAG: --client-ca-file="" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700414 2573 flags.go:64] FLAG: --cloud-provider="external" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700417 2573 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700421 2573 flags.go:64] FLAG: --cluster-domain="" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700424 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 19:55:12.706040 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700427 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700430 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700433 2573 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700442 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700445 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700448 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700451 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700454 2573 flags.go:64] FLAG: --contention-profiling="false" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700457 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700460 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700479 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700484 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700488 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700491 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700494 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700497 2573 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700500 2573 flags.go:64] FLAG: --enable-server="true" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700503 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700510 2573 flags.go:64] FLAG: --event-burst="100" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700513 2573 flags.go:64] FLAG: --event-qps="50" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700516 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700519 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700522 2573 flags.go:64] FLAG: --eviction-hard="" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700526 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 19:55:12.706688 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700529 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700532 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700549 2573 flags.go:64] FLAG: --eviction-soft="" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700553 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700556 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700560 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700563 2573 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700566 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700569 2573 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700572 2573 flags.go:64] FLAG: --feature-gates="" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700576 2573 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700579 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700582 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700585 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700588 2573 flags.go:64] FLAG: --healthz-port="10248" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700591 2573 flags.go:64] FLAG: --help="false" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700594 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-196.ec2.internal" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700598 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700601 2573 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700604 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700607 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700611 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700614 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 19:55:12.707307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700617 2573 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700620 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700623 2573 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700627 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700630 2573 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700633 2573 flags.go:64] FLAG: --kube-reserved="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700636 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700640 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700643 2573 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700646 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700649 2573 flags.go:64] FLAG: --lock-file="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700651 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700654 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700657 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700663 2573 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700666 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700669 2573 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700672 2573 flags.go:64] FLAG: --logging-format="text" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700675 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700678 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700681 2573 flags.go:64] FLAG: --manifest-url="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700684 2573 flags.go:64] FLAG: --manifest-url-header="" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700689 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700692 2573 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700702 2573 flags.go:64] FLAG: --max-pods="110" Sep 09 19:55:12.707960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700705 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700708 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700711 2573 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700718 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700721 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700724 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700727 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700735 2573 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700738 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700741 2573 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700745 2573 flags.go:64] FLAG: --pod-cidr="" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700748 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700753 2573 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700757 2573 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700762 2573 flags.go:64] FLAG: --pods-per-core="0" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700764 2573 flags.go:64] FLAG: --port="10250" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700768 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700771 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011a7de4abeb96afc" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700774 2573 flags.go:64] FLAG: --qos-reserved="" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700777 2573 flags.go:64] FLAG: --read-only-port="10255" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700780 2573 flags.go:64] FLAG: --register-node="true" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700783 2573 flags.go:64] FLAG: --register-with-taints="" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700786 2573 flags.go:64] FLAG: --registry-burst="10" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700790 2573 flags.go:64] FLAG: --registry-qps="5" Sep 09 19:55:12.708638 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700793 2573 flags.go:64] FLAG: --reserved-cpus="" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700796 2573 flags.go:64] FLAG: --reserved-memory="" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700800 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700803 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700806 2573 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700808 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700811 2573 flags.go:64] FLAG: --runonce="false" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700814 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700818 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700820 2573 flags.go:64] FLAG: --seccomp-default="false" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700823 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700826 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700829 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700832 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700835 2573 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700838 2573 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700841 2573 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700844 2573 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700847 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700851 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700854 2573 flags.go:64] FLAG: --system-cgroups="" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700857 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700864 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700867 2573 flags.go:64] FLAG: --tls-cert-file="" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700869 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 19:55:12.709337 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700873 2573 flags.go:64] FLAG: --tls-min-version="" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700876 2573 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700879 2573 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700882 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700885 2573 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700888 2573 flags.go:64] FLAG: --v="2" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700892 2573 flags.go:64] FLAG: --version="false" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700897 2573 flags.go:64] FLAG: --vmodule="" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700902 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.700905 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701029 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701034 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701037 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701040 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701043 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701046 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701049 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701052 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701055 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701057 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701060 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.710033 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701064 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701069 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701073 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701076 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701080 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701083 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701086 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701089 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701092 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701095 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701098 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701101 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701104 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701106 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701109 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701112 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701114 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701117 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701120 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701126 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.710614 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701129 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701132 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701135 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701137 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701140 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701143 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701145 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701148 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701150 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701153 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701156 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701158 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701161 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701164 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701166 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701169 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701172 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701177 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701180 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701183 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.711143 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701188 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701192 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701195 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701200 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701202 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701205 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701208 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701211 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701214 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701216 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701219 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701221 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701225 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701227 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701230 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701232 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701235 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701237 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701240 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701243 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.711681 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701245 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701248 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701250 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701253 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701255 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701258 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701260 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701263 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701265 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701269 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701272 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701275 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701279 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701281 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701284 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701286 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701289 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701291 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701293 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701306 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.712240 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701309 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701312 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701316 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701320 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701323 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701327 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701330 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.701333 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.701338 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.709539 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.709563 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.709578 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.709585 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709649 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.712801 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709656 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709660 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709664 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709668 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709671 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709674 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709677 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709680 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709683 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709686 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709688 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709691 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709694 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709697 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709700 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709702 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709705 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709707 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709709 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.713189 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709712 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709714 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709717 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709720 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709722 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709725 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709728 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709731 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709733 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709737 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709740 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709744 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709746 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709749 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709751 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709754 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709756 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709759 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709762 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709764 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.713707 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709767 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709769 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709772 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709774 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709777 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709779 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709782 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709785 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709787 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709790 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709792 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709795 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709797 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709800 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709802 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709805 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709808 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709810 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709813 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709815 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.714238 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709818 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709822 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709825 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709828 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709831 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709834 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709837 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709839 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709841 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709844 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709846 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709849 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709851 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709854 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709857 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709859 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709862 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709864 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709867 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.714772 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709871 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709875 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709877 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709880 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709883 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709885 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709888 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709891 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709893 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709895 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709898 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709901 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709903 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709905 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709908 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709913 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709917 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709920 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709923 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.715262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.709926 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.709932 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710070 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710074 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710077 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710080 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710083 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710086 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710088 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710091 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710094 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710096 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710099 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710101 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710104 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 19:55:12.715814 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710106 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710109 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710111 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710115 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710118 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710121 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710124 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710126 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710129 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710132 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710134 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710137 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710140 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710143 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710146 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710150 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710152 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710155 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710158 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710160 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 19:55:12.716217 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710163 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710166 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710168 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710171 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710173 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710176 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710178 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710180 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710183 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710185 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710188 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710190 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710192 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710195 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710198 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710200 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710202 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710205 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710207 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710209 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 19:55:12.716749 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710212 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710214 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710217 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710219 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710222 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710225 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710227 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710230 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710234 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710238 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710240 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710243 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710246 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710248 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710251 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710253 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710256 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710258 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710261 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710263 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 19:55:12.717275 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710265 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710268 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710270 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710273 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710275 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710278 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710280 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710283 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710285 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710287 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710291 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710294 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710297 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710299 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710301 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710304 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710307 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710309 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710312 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 19:55:12.717900 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710314 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710317 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710321 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710323 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710325 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710328 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:12.710330 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.710334 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.710496 2573 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.712975 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.714023 2573 server.go:1019] "Starting client certificate rotation" Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.714133 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.714205 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 19:55:12.718398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.717031 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 19:55:12.722319 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.722300 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 19:55:12.731576 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.731544 2573 log.go:25] "Validated CRI v1 runtime API" Sep 09 19:55:12.736848 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.736829 2573 log.go:25] "Validated CRI v1 image API" Sep 09 19:55:12.737812 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.737777 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 19:55:12.742274 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.742251 2573 fs.go:135] Filesystem UUIDs: map[1bdf0178-a2d5-48c0-bf1a-c48e65c1670e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f8b2ecfe-a5d1-4c9a-9671-c04b567b4909:/dev/nvme0n1p3] Sep 09 19:55:12.742349 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.742274 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 19:55:12.747915 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.747769 2573 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:12.746317622 +0000 UTC m=+0.351954359 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095999 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2056674bf157c3f960d121504d296e SystemUUID:ec205667-4bf1-57c3-f960-d121504d296e BootID:060282e8-0bba-4c42-98df-0c9f1d5b1aa3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d6:07:14:03:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d6:07:14:03:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:c8:4f:b7:4c:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 19:55:12.747915 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.747894 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 19:55:12.748092 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.748035 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 19:55:12.751663 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.751625 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 19:55:12.751814 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.751667 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 19:55:12.752067 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.752049 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 19:55:12.753245 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.753230 2573 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 19:55:12.753286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.753249 2573 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 19:55:12.753286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.753259 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 19:55:12.753903 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.753887 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 19:55:12.755680 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.755665 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 19:55:12.759096 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759078 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 09 19:55:12.759233 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759223 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 19:55:12.759264 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759258 2573 kubelet.go:486] "Attempting to sync node with API server" Sep 09 19:55:12.759293 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759273 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 19:55:12.759293 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759285 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 19:55:12.759371 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759296 2573 kubelet.go:398] "Adding apiserver pod source" Sep 09 19:55:12.759371 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759308 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 19:55:12.759795 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759783 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 19:55:12.759836 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.759803 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 19:55:12.762495 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.762460 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 19:55:12.765627 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765603 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 19:55:12.765700 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765651 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 19:55:12.765861 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765850 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 19:55:12.765902 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765869 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 19:55:12.765902 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765876 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 19:55:12.765967 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765915 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 19:55:12.765967 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765925 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 19:55:12.765967 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.765930 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 19:55:12.766511 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766501 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 19:55:12.766511 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766512 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 19:55:12.766584 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766519 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 19:55:12.766584 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766524 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 19:55:12.766584 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766538 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 19:55:12.766584 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.766545 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 19:55:12.767509 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.767498 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 19:55:12.767509 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.767509 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 19:55:12.771771 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.771743 2573 server.go:1268] "Started kubelet" Sep 09 19:55:12.771875 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.771797 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 19:55:12.772652 ip-10-0-141-196 systemd[1]: Started Kubernetes Kubelet. Sep 09 19:55:12.776555 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.776487 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 19:55:12.776651 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.776591 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 19:55:12.777805 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.777782 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 19:55:12.778213 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.778187 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 19:55:12.778435 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.778221 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 19:55:12.778521 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.778450 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 19:55:12.779060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.779041 2573 server.go:310] "Adding debug handlers to kubelet server" Sep 09 19:55:12.779223 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.779202 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 19:55:12.784729 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.783538 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-196.ec2.internal.18d3bf3554b79382 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-196.ec2.internal,UID:ip-10-0-141-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-196.ec2.internal,},FirstTimestamp:2026-09-09 19:55:12.771707778 +0000 UTC m=+0.377344514,LastTimestamp:2026-09-09 19:55:12.771707778 +0000 UTC m=+0.377344514,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-196.ec2.internal,}" Sep 09 19:55:12.786649 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.786630 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 19:55:12.786649 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.786643 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 19:55:12.786927 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.786764 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:12.787357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.787344 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 19:55:12.787481 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.787454 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 19:55:12.787597 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.787506 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 19:55:12.787706 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.787657 2573 reconstruct.go:97] "Volume reconstruction finished" Sep 09 19:55:12.787706 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.787666 2573 reconciler.go:29] "Reconciler: start to sync state" Sep 09 19:55:12.788831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.788816 2573 factory.go:153] Registering CRI-O factory Sep 09 19:55:12.788979 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.788960 2573 factory.go:223] Registration of the crio container factory successfully Sep 09 19:55:12.789073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789027 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 19:55:12.789073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789039 2573 factory.go:55] Registering systemd factory Sep 09 19:55:12.789073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789046 2573 factory.go:223] Registration of the systemd container factory successfully Sep 09 19:55:12.789073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789059 2573 factory.go:103] Registering Raw factory Sep 09 19:55:12.789073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789069 2573 manager.go:1196] Started watching for new ooms in manager Sep 09 19:55:12.789447 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.789435 2573 manager.go:349] Starting recovery of all containers Sep 09 19:55:12.795053 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.795030 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7z9rl" Sep 09 19:55:12.801407 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.801383 2573 manager.go:354] Recovery completed Sep 09 19:55:12.803785 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.803761 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 19:55:12.803785 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.803758 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 19:55:12.806352 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.806332 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:12.807271 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.807254 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7z9rl" Sep 09 19:55:12.808344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.808326 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:12.808415 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.808358 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:12.808415 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.808370 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:12.809069 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809053 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 19:55:12.809069 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809066 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 19:55:12.809185 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809079 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 09 19:55:12.809185 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809081 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:12.809822 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809805 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:12.809899 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809838 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:12.809899 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.809851 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:12.810063 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.810048 2573 policy_none.go:49] "None policy: Start" Sep 09 19:55:12.810137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.810066 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 19:55:12.810137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.810077 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 19:55:12.811052 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.811040 2573 policy_none.go:47] "Start" Sep 09 19:55:12.842434 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.842189 2573 manager.go:337] "Starting Device Plugin manager" Sep 09 19:55:12.842595 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.842498 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 19:55:12.842595 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.842514 2573 server.go:87] "Starting device plugin registration server" Sep 09 19:55:12.842860 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.842843 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 19:55:12.842912 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.842860 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 19:55:12.844812 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.843105 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 19:55:12.844812 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.843203 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 19:55:12.844812 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.843212 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 19:55:12.845654 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.845526 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 19:55:12.845654 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.845585 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:12.920157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.920122 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 19:55:12.921518 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.921496 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 19:55:12.921649 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.921553 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 19:55:12.921649 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.921580 2573 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 19:55:12.921748 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.921695 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 19:55:12.925281 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.925253 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 19:55:12.944009 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.943906 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:12.945071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.945048 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:12.945188 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.945086 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:12.945188 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.945098 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:12.945188 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.945122 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-196.ec2.internal" Sep 09 19:55:12.958271 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:12.958244 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-196.ec2.internal" Sep 09 19:55:12.958430 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:12.958279 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-196.ec2.internal\": node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.003592 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.003557 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.021781 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.021751 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal"] Sep 09 19:55:13.021865 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.021839 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:13.022893 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.022870 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:13.022994 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.022904 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:13.022994 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.022918 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:13.024009 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.023996 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:13.024125 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.024166 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024143 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:13.024848 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024829 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:13.024955 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024863 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:13.024955 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024879 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:13.024955 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024833 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:13.024955 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024907 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:13.024955 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.024916 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:13.025983 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.025967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.026030 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.026005 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 19:55:13.026763 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.026744 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientMemory" Sep 09 19:55:13.026845 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.026780 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 19:55:13.026845 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.026796 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeHasSufficientPID" Sep 09 19:55:13.046255 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.046226 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-196.ec2.internal\" not found" node="ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.050671 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.050649 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-196.ec2.internal\" not found" node="ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.089012 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.088970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.089012 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.089012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.089233 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.089036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/538be1d961f54997392b43e39784ee03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-196.ec2.internal\" (UID: \"538be1d961f54997392b43e39784ee03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.104163 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.104103 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.189253 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.189253 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.189508 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/538be1d961f54997392b43e39784ee03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-196.ec2.internal\" (UID: \"538be1d961f54997392b43e39784ee03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.189508 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.189508 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189325 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/538be1d961f54997392b43e39784ee03-config\") pod \"kube-apiserver-proxy-ip-10-0-141-196.ec2.internal\" (UID: \"538be1d961f54997392b43e39784ee03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.189508 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.189376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb90bd70394b1603af89cd10734898d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal\" (UID: \"cb90bd70394b1603af89cd10734898d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.205328 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.205257 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.305559 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.305527 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.348717 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.348679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.353662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.353640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:13.406328 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.406291 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.506998 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.506914 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.607581 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.607545 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.708113 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.708083 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.714273 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.714244 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 19:55:13.714431 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.714402 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 19:55:13.808892 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.808804 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:13 +0000 UTC" deadline="2028-05-06 11:30:15.569583579 +0000 UTC" Sep 09 19:55:13.808892 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.808846 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14511h35m1.760741724s" Sep 09 19:55:13.808892 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:13.808816 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-196.ec2.internal\" not found" Sep 09 19:55:13.810132 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:13.810117 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 19:55:14.961207 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:14.961176 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb90bd70394b1603af89cd10734898d1.slice/crio-b773ecf3d7e6e7bd1938c46e2809276429628569130276fa0c42f783d738749b WatchSource:0}: Error finding container b773ecf3d7e6e7bd1938c46e2809276429628569130276fa0c42f783d738749b: Status 404 returned error can't find the container with id b773ecf3d7e6e7bd1938c46e2809276429628569130276fa0c42f783d738749b Sep 09 19:55:14.964522 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:14.964505 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 19:55:14.973896 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:14.973864 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" Sep 09 19:55:14.985492 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:14.985465 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 19:55:14.987185 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:14.987170 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" Sep 09 19:55:14.995699 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:14.995672 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 19:55:15.012309 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.012256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" event={"ID":"cb90bd70394b1603af89cd10734898d1","Type":"ContainerStarted","Data":"b773ecf3d7e6e7bd1938c46e2809276429628569130276fa0c42f783d738749b"} Sep 09 19:55:15.183536 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.183454 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 19:55:15.401428 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.401395 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 19:55:15.847069 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.847030 2573 apiserver.go:52] "Watching apiserver" Sep 09 19:55:15.853694 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.853664 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 19:55:15.854098 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.854067 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-5n2pt","openshift-dns/node-resolver-klb6w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal","openshift-multus/multus-fncsn","openshift-multus/network-metrics-daemon-fq6p4","openshift-network-diagnostics/network-check-target-l6hvj","kube-system/global-pull-secret-syncer-wdk27","kube-system/konnectivity-agent-w2jsk","openshift-image-registry/node-ca-6tzfm","openshift-multus/multus-additional-cni-plugins-v9c24","openshift-network-operator/iptables-alerter-cf2rk","openshift-ovn-kubernetes/ovnkube-node-jjnht","kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j"] Sep 09 19:55:15.857020 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.856999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.859193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.859170 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.861107 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.861215 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861192 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fqtvb\"" Sep 09 19:55:15.861215 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861212 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.861787 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.861929 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861916 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.862013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.861949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jzvvv\"" Sep 09 19:55:15.862662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.862642 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.863669 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.863649 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rfvlk\"" Sep 09 19:55:15.863957 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.863938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:15.864086 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.864059 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:15.865734 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.865716 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.865889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.865869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:15.865977 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.865941 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:15.868805 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.868787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.868895 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.868860 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:15.869503 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.869482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.871073 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.871057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.872097 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.871657 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 19:55:15.873628 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.873599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.874736 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.874219 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 19:55:15.874736 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.874469 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 19:55:15.874979 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.874964 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 19:55:15.875354 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.875337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6zfh\"" Sep 09 19:55:15.875926 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.875905 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.878074 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.878048 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:15.880506 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.880484 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.881266 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.881249 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 19:55:15.882375 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.881647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.882375 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.881978 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.884496 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.884474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.885556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysconfig\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885655 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-tuned\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885655 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cnibin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.885655 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-netns\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpc6t\" (UniqueName: \"kubernetes.io/projected/fb7783cd-ee82-4563-8ad0-35b71502baa3-kube-api-access-kpc6t\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-systemd\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-sys\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-var-lib-kubelet\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-kubelet-config\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.885872 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-dbus\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d11c2e4-89ae-4d07-89d6-5e44a235076c-serviceca\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-kubernetes\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-system-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-conf-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.885996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-multus-certs\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-system-cni-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886042 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-run\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-socket-dir-parent\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-multus\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2csq\" (UniqueName: \"kubernetes.io/projected/7d11c2e4-89ae-4d07-89d6-5e44a235076c-kube-api-access-s2csq\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-kubelet\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-tmp\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886256 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-bin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-daemon-config\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.886334 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886323 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-etc-kubernetes\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72392e60-cc8c-4b7c-80cb-48664d636e08-konnectivity-ca\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-cnibin\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-conf\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cni-binary-copy\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-hostroot\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72392e60-cc8c-4b7c-80cb-48664d636e08-agent-certs\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d11c2e4-89ae-4d07-89d6-5e44a235076c-host\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-lib-modules\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7783cd-ee82-4563-8ad0-35b71502baa3-hosts-file\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjm5l\" (UniqueName: \"kubernetes.io/projected/e7efdaff-1e1a-46bb-8246-daec96e00667-kube-api-access-pjm5l\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-modprobe-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-host\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.887114 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9cs\" (UniqueName: \"kubernetes.io/projected/eac95add-8a4f-4c64-8b85-f67b97f612d2-kube-api-access-ml9cs\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.887828 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886759 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-k8s-cni-cncf-io\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.887828 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4ql\" (UniqueName: \"kubernetes.io/projected/1bd68d96-fd12-46c1-a848-8866cc49e8f3-kube-api-access-vw4ql\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.887828 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755zk\" (UniqueName: \"kubernetes.io/projected/68bbb879-fc6e-442b-b100-0ce397467ceb-kube-api-access-755zk\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:15.887828 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7783cd-ee82-4563-8ad0-35b71502baa3-tmp-dir\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.887828 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.886879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888282 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888336 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888358 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888367 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rqbsj\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lsb5l\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888362 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888582 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888593 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888610 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888663 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bc7pc\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888720 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888798 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888818 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888822 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.888858 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 19:55:15.889157 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.889075 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmv9x\"" Sep 09 19:55:15.889919 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.889425 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 19:55:15.889919 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.889536 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 19:55:15.900328 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.900303 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrdct\"" Sep 09 19:55:15.900479 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.900398 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 19:55:15.914707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.914676 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 19:55:15.937868 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.937837 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w6w2r" Sep 09 19:55:15.948624 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.948580 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w6w2r" Sep 09 19:55:15.975144 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.975108 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 19:55:15.987621 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7783cd-ee82-4563-8ad0-35b71502baa3-tmp-dir\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.987869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-slash\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.987869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysconfig\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.987869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-tuned\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.987869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysconfig\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.987869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cnibin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cnibin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-netns\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-netns\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d11c2e4-89ae-4d07-89d6-5e44a235076c-serviceca\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.987971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb7783cd-ee82-4563-8ad0-35b71502baa3-tmp-dir\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-system-cni-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.988102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-system-cni-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-kubernetes\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988213 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-kubernetes\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-system-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-multus-certs\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-run\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-socket-dir-parent\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-var-lib-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-system-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-socket-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-multus-certs\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-registration-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-run\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.988457 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-kubelet\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-tmp\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-socket-dir-parent\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-bin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-kubelet\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988508 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-daemon-config\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-bin\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72392e60-cc8c-4b7c-80cb-48664d636e08-konnectivity-ca\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-cnibin\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-kubelet\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988900 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d11c2e4-89ae-4d07-89d6-5e44a235076c-serviceca\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-cnibin\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.988979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-netd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-conf\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.989072 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cni-binary-copy\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72392e60-cc8c-4b7c-80cb-48664d636e08-agent-certs\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989108 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-conf\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d11c2e4-89ae-4d07-89d6-5e44a235076c-host\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjm5l\" (UniqueName: \"kubernetes.io/projected/e7efdaff-1e1a-46bb-8246-daec96e00667-kube-api-access-pjm5l\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-daemon-config\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765c05a-11e7-4fae-a7a1-9194360199a3-host-slash\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-env-overrides\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72392e60-cc8c-4b7c-80cb-48664d636e08-konnectivity-ca\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-systemd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-k8s-cni-cncf-io\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d11c2e4-89ae-4d07-89d6-5e44a235076c-host\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4ql\" (UniqueName: \"kubernetes.io/projected/1bd68d96-fd12-46c1-a848-8866cc49e8f3-kube-api-access-vw4ql\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989401 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-run-k8s-cni-cncf-io\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755zk\" (UniqueName: \"kubernetes.io/projected/68bbb879-fc6e-442b-b100-0ce397467ceb-kube-api-access-755zk\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.989830 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765c05a-11e7-4fae-a7a1-9194360199a3-iptables-alerter-script\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-systemd-units\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1bd68d96-fd12-46c1-a848-8866cc49e8f3-cni-binary-copy\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpc6t\" (UniqueName: \"kubernetes.io/projected/fb7783cd-ee82-4563-8ad0-35b71502baa3-kube-api-access-kpc6t\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-dbus\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-node-log\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65647a88-fc82-48e1-bf08-4312edc78fdd-ovn-node-metrics-cert\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-script-lib\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-systemd\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-sys\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.989885 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-var-lib-kubelet\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.990698 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-kubelet-config\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.990036 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.489985924 +0000 UTC m=+3.008865616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-var-lib-kubelet\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990098 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-binary-copy\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msr82\" (UniqueName: \"kubernetes.io/projected/65647a88-fc82-48e1-bf08-4312edc78fdd-kube-api-access-msr82\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990122 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-cni-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-conf-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990168 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-systemd\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcx7\" (UniqueName: \"kubernetes.io/projected/b765c05a-11e7-4fae-a7a1-9194360199a3-kube-api-access-7vcx7\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-sys-fs\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-multus-conf-dir\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990304 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-sysctl-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990036 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-kubelet-config\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.989945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-sys\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfp4\" (UniqueName: \"kubernetes.io/projected/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kube-api-access-dbfp4\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.991798 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-multus\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2csq\" (UniqueName: \"kubernetes.io/projected/7d11c2e4-89ae-4d07-89d6-5e44a235076c-kube-api-access-s2csq\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-etc-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-device-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-etc-kubernetes\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-bin\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-host-var-lib-cni-multus\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-hostroot\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cd560bd-2201-4588-a031-5c4ab028f42f-dbus\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-etc-kubernetes\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.990713 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990788 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1bd68d96-fd12-46c1-a848-8866cc49e8f3-hostroot\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:15.990829 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.490813059 +0000 UTC m=+3.009692765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:15.992630 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-config\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-lib-modules\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7783cd-ee82-4563-8ad0-35b71502baa3-hosts-file\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.990973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb7783cd-ee82-4563-8ad0-35b71502baa3-hosts-file\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-netns\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7efdaff-1e1a-46bb-8246-daec96e00667-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-lib-modules\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-ovn\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991135 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-log-socket\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-modprobe-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.991557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-host\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.992542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9cs\" (UniqueName: \"kubernetes.io/projected/eac95add-8a4f-4c64-8b85-f67b97f612d2-kube-api-access-ml9cs\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.992615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7efdaff-1e1a-46bb-8246-daec96e00667-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.992785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-modprobe-d\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.992855 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eac95add-8a4f-4c64-8b85-f67b97f612d2-host\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.993524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.993153 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72392e60-cc8c-4b7c-80cb-48664d636e08-agent-certs\") pod \"konnectivity-agent-w2jsk\" (UID: \"72392e60-cc8c-4b7c-80cb-48664d636e08\") " pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:15.994344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.993852 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-etc-tuned\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:15.996413 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:15.996392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eac95add-8a4f-4c64-8b85-f67b97f612d2-tmp\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:16.001002 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.000966 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:16.001002 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.000996 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:16.001180 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.001011 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:16.001180 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.001073 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:16.501053312 +0000 UTC m=+3.019933022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:16.003784 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.003742 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4ql\" (UniqueName: \"kubernetes.io/projected/1bd68d96-fd12-46c1-a848-8866cc49e8f3-kube-api-access-vw4ql\") pod \"multus-fncsn\" (UID: \"1bd68d96-fd12-46c1-a848-8866cc49e8f3\") " pod="openshift-multus/multus-fncsn" Sep 09 19:55:16.005466 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.005437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9cs\" (UniqueName: \"kubernetes.io/projected/eac95add-8a4f-4c64-8b85-f67b97f612d2-kube-api-access-ml9cs\") pod \"tuned-5n2pt\" (UID: \"eac95add-8a4f-4c64-8b85-f67b97f612d2\") " pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:16.005573 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.005483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjm5l\" (UniqueName: \"kubernetes.io/projected/e7efdaff-1e1a-46bb-8246-daec96e00667-kube-api-access-pjm5l\") pod \"multus-additional-cni-plugins-v9c24\" (UID: \"e7efdaff-1e1a-46bb-8246-daec96e00667\") " pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:16.005614 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.005578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755zk\" (UniqueName: \"kubernetes.io/projected/68bbb879-fc6e-442b-b100-0ce397467ceb-kube-api-access-755zk\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:16.006187 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.006166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpc6t\" (UniqueName: \"kubernetes.io/projected/fb7783cd-ee82-4563-8ad0-35b71502baa3-kube-api-access-kpc6t\") pod \"node-resolver-klb6w\" (UID: \"fb7783cd-ee82-4563-8ad0-35b71502baa3\") " pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:16.006888 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.006867 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2csq\" (UniqueName: \"kubernetes.io/projected/7d11c2e4-89ae-4d07-89d6-5e44a235076c-kube-api-access-s2csq\") pod \"node-ca-6tzfm\" (UID: \"7d11c2e4-89ae-4d07-89d6-5e44a235076c\") " pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:16.093616 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.093616 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msr82\" (UniqueName: \"kubernetes.io/projected/65647a88-fc82-48e1-bf08-4312edc78fdd-kube-api-access-msr82\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcx7\" (UniqueName: \"kubernetes.io/projected/b765c05a-11e7-4fae-a7a1-9194360199a3-kube-api-access-7vcx7\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-sys-fs\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfp4\" (UniqueName: \"kubernetes.io/projected/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kube-api-access-dbfp4\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-sys-fs\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.093788 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-etc-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-device-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-bin\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-etc-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-config\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093870 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-bin\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-netns\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093888 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-device-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-ovn\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-netns\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-log-socket\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093974 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.093996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-ovn\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-log-socket\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-slash\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094032 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-var-lib-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094066 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-slash\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-socket-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094149 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-etc-selinux\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-registration-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-kubelet\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094202 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-registration-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-var-lib-openvswitch\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-netd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-socket-dir\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094261 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-kubelet\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765c05a-11e7-4fae-a7a1-9194360199a3-host-slash\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094297 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-cni-netd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-env-overrides\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b765c05a-11e7-4fae-a7a1-9194360199a3-host-slash\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.094988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-systemd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765c05a-11e7-4fae-a7a1-9194360199a3-iptables-alerter-script\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-config\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094408 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-run-systemd\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-systemd-units\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094500 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-systemd-units\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-node-log\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65647a88-fc82-48e1-bf08-4312edc78fdd-ovn-node-metrics-cert\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-script-lib\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094594 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65647a88-fc82-48e1-bf08-4312edc78fdd-node-log\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-env-overrides\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b765c05a-11e7-4fae-a7a1-9194360199a3-iptables-alerter-script\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.095499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.094961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65647a88-fc82-48e1-bf08-4312edc78fdd-ovnkube-script-lib\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.096854 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.096839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65647a88-fc82-48e1-bf08-4312edc78fdd-ovn-node-metrics-cert\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.101882 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.101835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcx7\" (UniqueName: \"kubernetes.io/projected/b765c05a-11e7-4fae-a7a1-9194360199a3-kube-api-access-7vcx7\") pod \"iptables-alerter-cf2rk\" (UID: \"b765c05a-11e7-4fae-a7a1-9194360199a3\") " pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.101978 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.101926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msr82\" (UniqueName: \"kubernetes.io/projected/65647a88-fc82-48e1-bf08-4312edc78fdd-kube-api-access-msr82\") pod \"ovnkube-node-jjnht\" (UID: \"65647a88-fc82-48e1-bf08-4312edc78fdd\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.101978 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.101940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfp4\" (UniqueName: \"kubernetes.io/projected/23f21cd5-5ddb-4de4-bdeb-fa749af50e10-kube-api-access-dbfp4\") pod \"aws-ebs-csi-driver-node-6s22j\" (UID: \"23f21cd5-5ddb-4de4-bdeb-fa749af50e10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.118891 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.118864 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 19:55:16.168216 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.168174 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" Sep 09 19:55:16.175396 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.175369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klb6w" Sep 09 19:55:16.178907 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.178876 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac95add_8a4f_4c64_8b85_f67b97f612d2.slice/crio-48d1cd05c5ce66d13e7ce28bae1c69a131514a19426b6581c53072f3a56d512f WatchSource:0}: Error finding container 48d1cd05c5ce66d13e7ce28bae1c69a131514a19426b6581c53072f3a56d512f: Status 404 returned error can't find the container with id 48d1cd05c5ce66d13e7ce28bae1c69a131514a19426b6581c53072f3a56d512f Sep 09 19:55:16.185545 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.185522 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fncsn" Sep 09 19:55:16.185824 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.185747 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7783cd_ee82_4563_8ad0_35b71502baa3.slice/crio-5aceb82cccd4a0993286eb096100d46b36a60ff2d20e8f8aaa8972e1d8b27fee WatchSource:0}: Error finding container 5aceb82cccd4a0993286eb096100d46b36a60ff2d20e8f8aaa8972e1d8b27fee: Status 404 returned error can't find the container with id 5aceb82cccd4a0993286eb096100d46b36a60ff2d20e8f8aaa8972e1d8b27fee Sep 09 19:55:16.193719 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.193700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:16.195982 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.195947 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd68d96_fd12_46c1_a848_8866cc49e8f3.slice/crio-13a1292cebaabde642593bd45aef30a7d2449bfed6dc8d863c002013b8613c25 WatchSource:0}: Error finding container 13a1292cebaabde642593bd45aef30a7d2449bfed6dc8d863c002013b8613c25: Status 404 returned error can't find the container with id 13a1292cebaabde642593bd45aef30a7d2449bfed6dc8d863c002013b8613c25 Sep 09 19:55:16.199299 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.199277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6tzfm" Sep 09 19:55:16.203819 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.203783 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72392e60_cc8c_4b7c_80cb_48664d636e08.slice/crio-91a67ca4fd88cb05b3c2efa5840a6f1655e31eb62eed1628030a21e77a650ef4 WatchSource:0}: Error finding container 91a67ca4fd88cb05b3c2efa5840a6f1655e31eb62eed1628030a21e77a650ef4: Status 404 returned error can't find the container with id 91a67ca4fd88cb05b3c2efa5840a6f1655e31eb62eed1628030a21e77a650ef4 Sep 09 19:55:16.206161 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.206136 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v9c24" Sep 09 19:55:16.215579 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.215366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cf2rk" Sep 09 19:55:16.217357 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.217328 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7efdaff_1e1a_46bb_8246_daec96e00667.slice/crio-7e25d14df05fd28bbe986ce08238affcd1fbb1ab7da214d005e98f55b592929f WatchSource:0}: Error finding container 7e25d14df05fd28bbe986ce08238affcd1fbb1ab7da214d005e98f55b592929f: Status 404 returned error can't find the container with id 7e25d14df05fd28bbe986ce08238affcd1fbb1ab7da214d005e98f55b592929f Sep 09 19:55:16.222485 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.222257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:16.226731 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.226708 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb765c05a_11e7_4fae_a7a1_9194360199a3.slice/crio-f3b6798a0ea0db3166562f4c66872d302fba546b243b9bc3c73a2e9f46b002cb WatchSource:0}: Error finding container f3b6798a0ea0db3166562f4c66872d302fba546b243b9bc3c73a2e9f46b002cb: Status 404 returned error can't find the container with id f3b6798a0ea0db3166562f4c66872d302fba546b243b9bc3c73a2e9f46b002cb Sep 09 19:55:16.226823 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.226800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" Sep 09 19:55:16.233519 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.233491 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65647a88_fc82_48e1_bf08_4312edc78fdd.slice/crio-fa385a93f17c80b23275d953a986a8ea9cb82bc02d1313bfb9a1b32bd8e25a26 WatchSource:0}: Error finding container fa385a93f17c80b23275d953a986a8ea9cb82bc02d1313bfb9a1b32bd8e25a26: Status 404 returned error can't find the container with id fa385a93f17c80b23275d953a986a8ea9cb82bc02d1313bfb9a1b32bd8e25a26 Sep 09 19:55:16.236763 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.236735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f21cd5_5ddb_4de4_bdeb_fa749af50e10.slice/crio-738dbd45a28db23b3dacf806d2fe8f366b82173c62ce3274da5d296e2289aaaf WatchSource:0}: Error finding container 738dbd45a28db23b3dacf806d2fe8f366b82173c62ce3274da5d296e2289aaaf: Status 404 returned error can't find the container with id 738dbd45a28db23b3dacf806d2fe8f366b82173c62ce3274da5d296e2289aaaf Sep 09 19:55:16.497473 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.497426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:16.497473 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.497479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:16.497739 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.497583 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:16.497739 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.497588 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:16.497739 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.497637 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.49762339 +0000 UTC m=+4.016503081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:16.497739 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.497650 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.497644401 +0000 UTC m=+4.016524093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:16.598591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.598557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:16.598780 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.598734 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:16.598780 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.598754 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:16.598780 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.598763 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:16.598884 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:16.598820 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.598803635 +0000 UTC m=+4.117683544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:16.775016 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:55:16.773965 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538be1d961f54997392b43e39784ee03.slice/crio-1171b719a7f96b11a7479807593214cba74568876cf76a2a3219d87df947c07b WatchSource:0}: Error finding container 1171b719a7f96b11a7479807593214cba74568876cf76a2a3219d87df947c07b: Status 404 returned error can't find the container with id 1171b719a7f96b11a7479807593214cba74568876cf76a2a3219d87df947c07b Sep 09 19:55:16.952459 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.952375 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:15 +0000 UTC" deadline="2028-03-02 12:46:47.044747613 +0000 UTC" Sep 09 19:55:16.952459 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:16.952449 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12952h51m30.092302721s" Sep 09 19:55:17.025472 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.025379 2573 generic.go:356] "Generic (PLEG): container finished" podID="cb90bd70394b1603af89cd10734898d1" containerID="681102ed68db247e1d53d9549da18ed9941ab89cd9a25426434e5fa1c0fcbb9b" exitCode=0 Sep 09 19:55:17.026047 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.026014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" event={"ID":"cb90bd70394b1603af89cd10734898d1","Type":"ContainerDied","Data":"681102ed68db247e1d53d9549da18ed9941ab89cd9a25426434e5fa1c0fcbb9b"} Sep 09 19:55:17.034270 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.034184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" event={"ID":"538be1d961f54997392b43e39784ee03","Type":"ContainerStarted","Data":"1171b719a7f96b11a7479807593214cba74568876cf76a2a3219d87df947c07b"} Sep 09 19:55:17.044707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.044668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" event={"ID":"23f21cd5-5ddb-4de4-bdeb-fa749af50e10","Type":"ContainerStarted","Data":"738dbd45a28db23b3dacf806d2fe8f366b82173c62ce3274da5d296e2289aaaf"} Sep 09 19:55:17.048967 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.048943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"fa385a93f17c80b23275d953a986a8ea9cb82bc02d1313bfb9a1b32bd8e25a26"} Sep 09 19:55:17.061345 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.061214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerStarted","Data":"7e25d14df05fd28bbe986ce08238affcd1fbb1ab7da214d005e98f55b592929f"} Sep 09 19:55:17.069566 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.069524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6tzfm" event={"ID":"7d11c2e4-89ae-4d07-89d6-5e44a235076c","Type":"ContainerStarted","Data":"c5d224a4b5e101b39d5d842fa8b7821459df606f93e8add9b22723e0ece33d53"} Sep 09 19:55:17.080537 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.080498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2jsk" event={"ID":"72392e60-cc8c-4b7c-80cb-48664d636e08","Type":"ContainerStarted","Data":"91a67ca4fd88cb05b3c2efa5840a6f1655e31eb62eed1628030a21e77a650ef4"} Sep 09 19:55:17.086539 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.086495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fncsn" event={"ID":"1bd68d96-fd12-46c1-a848-8866cc49e8f3","Type":"ContainerStarted","Data":"13a1292cebaabde642593bd45aef30a7d2449bfed6dc8d863c002013b8613c25"} Sep 09 19:55:17.090779 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.090739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cf2rk" event={"ID":"b765c05a-11e7-4fae-a7a1-9194360199a3","Type":"ContainerStarted","Data":"f3b6798a0ea0db3166562f4c66872d302fba546b243b9bc3c73a2e9f46b002cb"} Sep 09 19:55:17.092377 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.092349 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klb6w" event={"ID":"fb7783cd-ee82-4563-8ad0-35b71502baa3","Type":"ContainerStarted","Data":"5aceb82cccd4a0993286eb096100d46b36a60ff2d20e8f8aaa8972e1d8b27fee"} Sep 09 19:55:17.093809 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.093777 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" event={"ID":"eac95add-8a4f-4c64-8b85-f67b97f612d2","Type":"ContainerStarted","Data":"48d1cd05c5ce66d13e7ce28bae1c69a131514a19426b6581c53072f3a56d512f"} Sep 09 19:55:17.508878 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.508835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:17.509053 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.508919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:17.509125 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.509084 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:17.509165 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.509149 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:19.509130826 +0000 UTC m=+6.028010523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:17.509585 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.509565 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:17.509681 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.509622 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:19.509605504 +0000 UTC m=+6.028485209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:17.610120 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.609960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:17.610314 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.610167 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:17.610314 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.610189 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:17.610314 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.610202 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:17.610491 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:17.610339 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:19.610318674 +0000 UTC m=+6.129198368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:17.953665 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.953504 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:15 +0000 UTC" deadline="2028-05-18 12:31:06.516347864 +0000 UTC" Sep 09 19:55:17.953665 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:17.953585 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14800h35m48.562768466s" Sep 09 19:55:18.011353 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:18.011321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:18.011538 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:18.011498 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:18.012060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:18.012028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:18.012179 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:18.012158 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:18.012260 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:18.012243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:18.012332 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:18.012313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:18.112715 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:18.112674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" event={"ID":"cb90bd70394b1603af89cd10734898d1","Type":"ContainerStarted","Data":"c5ae5a5598479c83a88067714a8621c5511e94f4a8a2403d8978012aba60f4ff"} Sep 09 19:55:18.131902 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:18.131839 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-196.ec2.internal" podStartSLOduration=4.131797259 podStartE2EDuration="4.131797259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:18.130934056 +0000 UTC m=+4.649813770" watchObservedRunningTime="2026-09-09 19:55:18.131797259 +0000 UTC m=+4.650676973" Sep 09 19:55:19.525602 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:19.525346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:19.525602 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:19.525433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:19.525602 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.525512 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:19.525602 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.525512 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:19.525602 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.525578 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:23.525561376 +0000 UTC m=+10.044441082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:19.526246 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.525621 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:23.525592673 +0000 UTC m=+10.044472368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:19.626776 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:19.626166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:19.626776 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.626352 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:19.626776 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.626375 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:19.626776 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.626388 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:19.626776 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:19.626453 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:23.626435388 +0000 UTC m=+10.145315085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:20.010890 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:20.010806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:20.011034 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:20.010962 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:20.011376 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:20.011354 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:20.011496 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:20.011457 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:20.011623 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:20.011525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:20.011623 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:20.011591 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:22.009461 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:22.009428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:22.009842 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:22.009572 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:22.009842 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:22.009797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:22.009960 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:22.009923 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:22.010042 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:22.010023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:22.010146 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:22.010126 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:23.560956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:23.561032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.561154 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.561158 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.561219 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.561199507 +0000 UTC m=+18.080079204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:23.561283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.561257 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.561246709 +0000 UTC m=+18.080126406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:23.661619 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:23.661568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:23.661804 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.661770 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:23.661804 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.661790 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:23.661804 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.661804 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:23.661962 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:23.661865 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.661847578 +0000 UTC m=+18.180727285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:24.009432 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:24.009347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:24.009620 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:24.009497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:24.009808 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:24.009792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:24.009873 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:24.009859 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:24.009913 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:24.009896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:24.009946 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:24.009937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:26.009119 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:26.009080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:26.009763 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:26.009137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:26.009763 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:26.009148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:26.009763 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:26.009257 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:26.009763 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:26.009349 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:26.009763 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:26.009427 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:28.008915 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:28.008875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:28.009409 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:28.008885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:28.009409 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:28.009016 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:28.009409 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:28.009125 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:28.009409 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:28.008885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:28.009409 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:28.009198 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:30.008834 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:30.008793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:30.009283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:30.008796 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:30.009283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:30.008950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:30.009283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:30.008928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:30.009283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:30.009056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:30.009283 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:30.009128 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:31.625776 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:31.625740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:31.626333 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:31.625792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:31.626333 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.625890 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:31.626333 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.625890 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:31.626333 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.625960 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.625939652 +0000 UTC m=+34.144819356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:31.626333 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.625987 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.625966139 +0000 UTC m=+34.144845840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:31.726350 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:31.726307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:31.726548 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.726521 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:31.726619 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.726555 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:31.726619 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.726569 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:31.726697 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:31.726634 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.726615443 +0000 UTC m=+34.245495151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:32.009601 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:32.009517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:32.009762 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:32.009522 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:32.009762 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:32.009642 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:32.009762 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:32.009524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:32.009762 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:32.009730 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:32.009940 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:32.009820 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:34.009942 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:34.009903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:34.010451 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:34.010028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:34.010451 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:34.010058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:34.010451 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:34.010130 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:34.010451 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:34.010023 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:34.010451 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:34.010253 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:36.009495 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.009151 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:36.009961 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.009162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:36.009961 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:36.009587 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:36.009961 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:36.009671 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:36.009961 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.009212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:36.009961 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:36.009773 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:36.156336 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.156247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klb6w" event={"ID":"fb7783cd-ee82-4563-8ad0-35b71502baa3","Type":"ContainerStarted","Data":"016ffb8577dd110cff02f20f06995ab68986db5504a5f0c3951e8a35c7d0dec9"} Sep 09 19:55:36.158705 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.158347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" event={"ID":"eac95add-8a4f-4c64-8b85-f67b97f612d2","Type":"ContainerStarted","Data":"8b246f0841003d33f0ae1d9fe5b0a09dc02ac3bc8f08434284700795ec6c8d5d"} Sep 09 19:55:36.160086 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.160054 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" event={"ID":"538be1d961f54997392b43e39784ee03","Type":"ContainerStarted","Data":"243fcec629b06d17562ccda68090d2dbddc370ce8f094edf950bc1ae0392cca7"} Sep 09 19:55:36.162045 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.162020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" event={"ID":"23f21cd5-5ddb-4de4-bdeb-fa749af50e10","Type":"ContainerStarted","Data":"91b2270215865b3f740a4ab65bb0012bdda70e861c0e19eae8339db7c6f41db2"} Sep 09 19:55:36.166439 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166413 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"6b8ee56e7ec4bfffb094e9372100d3463fc5f49676f3b7e1b36f0d5b776084fa"} Sep 09 19:55:36.166561 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"ea6a83ceca0ac7bd54c6ae6a77177583adf431e02686c33584dfbe63100d393b"} Sep 09 19:55:36.166561 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"1257ed61bd964b0f20a018623045d6ad9b9b5deed2a16c683dca54f569f92388"} Sep 09 19:55:36.166672 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166591 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"1a71421f36b03cf20c2a0f435e7f07b93129c100fcfdbd981bfe81afa9c5f19b"} Sep 09 19:55:36.166672 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166641 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"599225f449d214b28924face426ea9b6db7c66f2ffbe3e88d6c713c3cebe5271"} Sep 09 19:55:36.166761 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.166690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"a666f6f675dd58b28e8189fc9292030e499e1353c920b52710bc32b661a115e4"} Sep 09 19:55:36.168306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.168279 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="cdc7c93a33dc14704d2b06eaf3f665aab472a0f18ec5a34327e229a71ac00a37" exitCode=0 Sep 09 19:55:36.168470 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.168366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"cdc7c93a33dc14704d2b06eaf3f665aab472a0f18ec5a34327e229a71ac00a37"} Sep 09 19:55:36.169868 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.169844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6tzfm" event={"ID":"7d11c2e4-89ae-4d07-89d6-5e44a235076c","Type":"ContainerStarted","Data":"19b5587abc0e16f97f98390c799d4f5bd88ae7f3f20e1992a5347ce17be6dc70"} Sep 09 19:55:36.171761 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.171737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2jsk" event={"ID":"72392e60-cc8c-4b7c-80cb-48664d636e08","Type":"ContainerStarted","Data":"7e774bcb569e3e1830f92235dac8079b3b22e526ef300c8966fc6ce0f989088d"} Sep 09 19:55:36.172146 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.172104 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-klb6w" podStartSLOduration=3.098989032 podStartE2EDuration="22.172088208s" totalImagesPullingTime="19.073099176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.171455109 +0000 UTC m=+22.690334823" watchObservedRunningTime="2026-09-09 19:55:36.172088208 +0000 UTC m=+22.690967921" Sep 09 19:55:36.173809 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.173787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fncsn" event={"ID":"1bd68d96-fd12-46c1-a848-8866cc49e8f3","Type":"ContainerStarted","Data":"f85c91d7303bbdaabb33bcc1da6d13d1dc13f510c6f91ab8001862b3200c9cd3"} Sep 09 19:55:36.187917 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.187862 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6tzfm" podStartSLOduration=3.138663178 podStartE2EDuration="22.187843628s" totalImagesPullingTime="19.04918045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.187542353 +0000 UTC m=+22.706422070" watchObservedRunningTime="2026-09-09 19:55:36.187843628 +0000 UTC m=+22.706723343" Sep 09 19:55:36.265172 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.265113 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-196.ec2.internal" podStartSLOduration=22.265094946 podStartE2EDuration="22.265094946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.245206247 +0000 UTC m=+22.764085962" watchObservedRunningTime="2026-09-09 19:55:36.265094946 +0000 UTC m=+22.783974656" Sep 09 19:55:36.265345 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.265302 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5n2pt" podStartSLOduration=3.168297645 podStartE2EDuration="22.265293327s" totalImagesPullingTime="19.096995682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.264934782 +0000 UTC m=+22.783814496" watchObservedRunningTime="2026-09-09 19:55:36.265293327 +0000 UTC m=+22.784173039" Sep 09 19:55:36.297345 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.297297 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fncsn" podStartSLOduration=3.087189195 podStartE2EDuration="22.297278247s" totalImagesPullingTime="19.210089052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.281436976 +0000 UTC m=+22.800316690" watchObservedRunningTime="2026-09-09 19:55:36.297278247 +0000 UTC m=+22.816157960" Sep 09 19:55:36.297664 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.297592 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w2jsk" podStartSLOduration=4.298667963 podStartE2EDuration="22.297574694s" totalImagesPullingTime="17.998906731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:36.297075996 +0000 UTC m=+22.815955708" watchObservedRunningTime="2026-09-09 19:55:36.297574694 +0000 UTC m=+22.816454409" Sep 09 19:55:36.601263 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.601181 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 19:55:36.942985 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.942871 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:36.601253112Z","UUID":"6d609c2f-7429-48d6-86f8-0a722cc36457","Handler":null,"Name":"","Endpoint":""} Sep 09 19:55:36.944568 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.944545 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 19:55:36.944691 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:36.944586 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 19:55:37.179587 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:37.179501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" event={"ID":"23f21cd5-5ddb-4de4-bdeb-fa749af50e10","Type":"ContainerStarted","Data":"72415dcf6f04b996620d2a0e372c871da2a86ba3bde227fb8795f74569afedff"} Sep 09 19:55:37.181861 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:37.181836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cf2rk" event={"ID":"b765c05a-11e7-4fae-a7a1-9194360199a3","Type":"ContainerStarted","Data":"5b2a2f013976e1363b10e0ebf023e5d5fe320f18eb9be1f82ed687b44277405e"} Sep 09 19:55:37.198705 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:37.198647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cf2rk" podStartSLOduration=4.020126098 podStartE2EDuration="23.19862572s" totalImagesPullingTime="19.178499622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.197975893 +0000 UTC m=+23.716855606" watchObservedRunningTime="2026-09-09 19:55:37.19862572 +0000 UTC m=+23.717505434" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.008621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.008616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:38.008750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.008617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:38.008817 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:38.010553 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:38.008938 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:38.186896 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.186768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" event={"ID":"23f21cd5-5ddb-4de4-bdeb-fa749af50e10","Type":"ContainerStarted","Data":"22797e999a031d6785b1f4c3aaea420b09ec72dc4b09a2fd7879195e71365c58"} Sep 09 19:55:38.191747 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.191717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"81e789a25c614afff00c19e46201079e5f431b592c61cffee27b9e8e8857dfaa"} Sep 09 19:55:38.208101 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:38.208029 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6s22j" podStartSLOduration=3.420356772 podStartE2EDuration="24.208003465s" totalImagesPullingTime="20.787646693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:38.207645429 +0000 UTC m=+24.726525143" watchObservedRunningTime="2026-09-09 19:55:38.208003465 +0000 UTC m=+24.726883180" Sep 09 19:55:40.009178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.009137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:40.009815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.009137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:40.009815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.009301 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:40.009815 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:40.009317 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:40.009815 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:40.009420 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:40.009815 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:40.009552 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:40.115654 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.115624 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klb6w_fb7783cd-ee82-4563-8ad0-35b71502baa3/dns-node-resolver/0.log" Sep 09 19:55:40.619505 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.619255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:40.619947 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.619926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:40.894805 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:40.894731 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6tzfm_7d11c2e4-89ae-4d07-89d6-5e44a235076c/node-ca/0.log" Sep 09 19:55:41.201610 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.201513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" event={"ID":"65647a88-fc82-48e1-bf08-4312edc78fdd","Type":"ContainerStarted","Data":"ef4761aa050267c2d2e7ee5a3907f4adf5a5381426193c1c60a315c66a26e813"} Sep 09 19:55:41.202077 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.201781 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:41.202077 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.201890 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:41.202077 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.201937 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:41.204243 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.204199 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="5162443f0432227d3ba56999870438803a8a96c816308f77599a5a49308482f6" exitCode=0 Sep 09 19:55:41.204428 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.204331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"5162443f0432227d3ba56999870438803a8a96c816308f77599a5a49308482f6"} Sep 09 19:55:41.204581 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.204562 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:41.205921 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.205100 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w2jsk" Sep 09 19:55:41.219099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.219077 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:41.222716 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.222699 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:55:41.233905 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:41.233863 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" podStartSLOduration=7.887509223 podStartE2EDuration="27.233849891s" totalImagesPullingTime="19.346340668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:41.232449221 +0000 UTC m=+27.751328971" watchObservedRunningTime="2026-09-09 19:55:41.233849891 +0000 UTC m=+27.752729604" Sep 09 19:55:42.009740 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.009562 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:42.009897 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.009837 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:42.009897 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.009711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:42.010022 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.009942 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:42.010022 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.009593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:42.010127 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.010035 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:42.209019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.208937 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="de82c204a4b806d15c3792a253eea313a4ff55c93cc15d1030d9accaac4a52cb" exitCode=0 Sep 09 19:55:42.209459 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.209025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"de82c204a4b806d15c3792a253eea313a4ff55c93cc15d1030d9accaac4a52cb"} Sep 09 19:55:42.238604 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.238572 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fq6p4"] Sep 09 19:55:42.238777 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.238684 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:42.238827 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.238798 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:42.242199 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.242175 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdk27"] Sep 09 19:55:42.242366 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.242291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:42.242430 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.242397 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:42.242726 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.242700 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6hvj"] Sep 09 19:55:42.242836 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:42.242779 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:42.242889 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:42.242869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:44.009331 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:44.009289 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:44.009960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:44.009392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:44.009960 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:44.009506 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:44.009960 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:44.009406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:44.009960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:44.009561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:44.009960 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:44.009611 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:45.218482 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:45.218279 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="7bbb68f798e5f5ffc15eb50a6646b9b147bb181aa6e3871afc9bc36e33ff17bc" exitCode=0 Sep 09 19:55:45.219042 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:45.218361 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"7bbb68f798e5f5ffc15eb50a6646b9b147bb181aa6e3871afc9bc36e33ff17bc"} Sep 09 19:55:46.009039 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:46.009003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:46.009039 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:46.009025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:46.009319 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:46.009012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:46.009319 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:46.009144 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:46.009399 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:46.009311 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:46.009441 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:46.009393 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:47.640589 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:47.640548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:47.641083 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:47.640614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:47.641083 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.640715 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:47.641083 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.640732 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:47.641083 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.640799 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret podName:0cd560bd-2201-4588-a031-5c4ab028f42f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.640777396 +0000 UTC m=+66.159657102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret") pod "global-pull-secret-syncer-wdk27" (UID: "0cd560bd-2201-4588-a031-5c4ab028f42f") : object "kube-system"/"original-pull-secret" not registered Sep 09 19:55:47.641083 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.640820 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs podName:68bbb879-fc6e-442b-b100-0ce397467ceb nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.640809818 +0000 UTC m=+66.159689515 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs") pod "network-metrics-daemon-fq6p4" (UID: "68bbb879-fc6e-442b-b100-0ce397467ceb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 19:55:47.741019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:47.740956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:47.741194 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.741143 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 19:55:47.741194 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.741166 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 19:55:47.741194 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.741176 2573 projected.go:196] Error preparing data for projected volume kube-api-access-2fkhv for pod openshift-network-diagnostics/network-check-target-l6hvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:47.741346 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:47.741250 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv podName:00e74531-8617-4f73-b0f6-ad325c79086f nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.741217254 +0000 UTC m=+66.260096949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2fkhv" (UniqueName: "kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv") pod "network-check-target-l6hvj" (UID: "00e74531-8617-4f73-b0f6-ad325c79086f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 19:55:48.009289 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:48.009177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:48.009289 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:48.009188 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:48.009556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:48.009377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:48.009556 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:48.009382 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:48.009556 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:48.009477 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:48.009702 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:48.009598 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:50.009382 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:50.009340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:50.009382 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:50.009349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:50.009922 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:50.009484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:50.009922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:50.009530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:50.009922 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:50.009608 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:50.009922 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:50.009668 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:50.233757 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:50.233663 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="a607b3288d370957e5bd2f4b64c0cbfdb320865872996346111beefa1b02d837" exitCode=0 Sep 09 19:55:50.233757 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:50.233727 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"a607b3288d370957e5bd2f4b64c0cbfdb320865872996346111beefa1b02d837"} Sep 09 19:55:51.239095 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:51.239063 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7efdaff-1e1a-46bb-8246-daec96e00667" containerID="b35dfc22d6a83c6f346d124c2d32ca79d3bad0b1b564eea327411cabf64a4b2e" exitCode=0 Sep 09 19:55:51.239487 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:51.239111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerDied","Data":"b35dfc22d6a83c6f346d124c2d32ca79d3bad0b1b564eea327411cabf64a4b2e"} Sep 09 19:55:52.009017 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:52.008980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:52.009168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:52.008992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:52.009168 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:52.009100 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:52.009290 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:52.009195 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:52.009290 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:52.008992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:52.009366 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:52.009317 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:52.244716 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:52.244678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v9c24" event={"ID":"e7efdaff-1e1a-46bb-8246-daec96e00667","Type":"ContainerStarted","Data":"bef6844deed0b5ad3f131a96cea126341103dd2dc2ba2b029b7e87229ca3a775"} Sep 09 19:55:52.274668 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:52.274616 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v9c24" podStartSLOduration=7.093529033 podStartE2EDuration="38.274599928s" totalImagesPullingTime="31.181070895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:52.273782995 +0000 UTC m=+38.792662699" watchObservedRunningTime="2026-09-09 19:55:52.274599928 +0000 UTC m=+38.793479640" Sep 09 19:55:54.009998 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:54.009958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:54.010496 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:54.010051 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:54.010496 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:54.010092 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:54.010496 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:54.010093 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:54.010496 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:54.010200 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:54.010496 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:54.010318 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:56.008784 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:56.008746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:56.008784 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:56.008782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:56.009217 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:56.008753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:56.009217 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:56.008886 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:56.009217 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:56.008931 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:56.009217 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:56.009035 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:55:58.009198 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:58.009156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:55:58.009606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:58.009159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:55:58.009606 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:58.009327 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:55:58.009606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:55:58.009159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:55:58.009606 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:58.009359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:55:58.009606 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:55:58.009428 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:56:00.009172 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:00.009136 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:00.009172 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:00.009140 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:00.009585 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:00.009300 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:56:00.009585 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:00.009351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:00.009585 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:00.009435 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:56:00.009585 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:00.009498 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:56:02.009280 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.009253 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:02.009711 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.009279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:02.009711 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.009289 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:02.009711 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:02.009352 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdk27" podUID="0cd560bd-2201-4588-a031-5c4ab028f42f" Sep 09 19:56:02.009711 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:02.009412 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fq6p4" podUID="68bbb879-fc6e-442b-b100-0ce397467ceb" Sep 09 19:56:02.009711 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:02.009512 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6hvj" podUID="00e74531-8617-4f73-b0f6-ad325c79086f" Sep 09 19:56:02.211553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.211525 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-196.ec2.internal" event="NodeReady" Sep 09 19:56:02.211708 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.211652 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 19:56:02.282710 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.282615 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9vzms"] Sep 09 19:56:02.305924 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.305887 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5fp5p"] Sep 09 19:56:02.306121 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.306000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.308500 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.308478 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 19:56:02.308637 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.308526 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5xtg\"" Sep 09 19:56:02.308637 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.308557 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 19:56:02.329731 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.329700 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5fp5p"] Sep 09 19:56:02.329731 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.329735 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9vzms"] Sep 09 19:56:02.329974 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.329841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.332219 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.332201 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 19:56:02.332479 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.332464 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 19:56:02.332568 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.332504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmppz\"" Sep 09 19:56:02.332627 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.332591 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 19:56:02.368368 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.368337 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4z8tv"] Sep 09 19:56:02.387689 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.387661 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.390130 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4z8tv"] Sep 09 19:56:02.390592 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390560 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 19:56:02.390592 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390576 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 19:56:02.390760 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390607 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 19:56:02.390760 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390635 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhnk7\"" Sep 09 19:56:02.390842 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.390824 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 19:56:02.449797 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx9m\" (UniqueName: \"kubernetes.io/projected/35286300-d6c0-42e7-83f2-9c9ecec02a87-kube-api-access-gbx9m\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.449797 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-tmp-dir\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.450005 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-config-volume\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.450005 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35286300-d6c0-42e7-83f2-9c9ecec02a87-cert\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.450005 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-metrics-tls\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.450005 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.449947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsdx\" (UniqueName: \"kubernetes.io/projected/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-kube-api-access-znsdx\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.550401 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsdx\" (UniqueName: \"kubernetes.io/projected/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-kube-api-access-znsdx\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.550542 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scx8z\" (UniqueName: \"kubernetes.io/projected/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-api-access-scx8z\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.550542 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx9m\" (UniqueName: \"kubernetes.io/projected/35286300-d6c0-42e7-83f2-9c9ecec02a87-kube-api-access-gbx9m\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.550542 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.550542 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-tmp-dir\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.550680 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-config-volume\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.550680 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35286300-d6c0-42e7-83f2-9c9ecec02a87-cert\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.550741 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-metrics-tls\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.550776 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-crio-socket\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.550776 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-data-volume\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.550864 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.550918 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.550905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-tmp-dir\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.551133 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.551116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-config-volume\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.554985 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.554943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-metrics-tls\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.554985 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.554973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35286300-d6c0-42e7-83f2-9c9ecec02a87-cert\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.558059 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.558029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx9m\" (UniqueName: \"kubernetes.io/projected/35286300-d6c0-42e7-83f2-9c9ecec02a87-kube-api-access-gbx9m\") pod \"ingress-canary-5fp5p\" (UID: \"35286300-d6c0-42e7-83f2-9c9ecec02a87\") " pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.558177 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.558100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsdx\" (UniqueName: \"kubernetes.io/projected/5b2043cc-0dd5-48b3-b0a8-4681c68594dd-kube-api-access-znsdx\") pod \"dns-default-9vzms\" (UID: \"5b2043cc-0dd5-48b3-b0a8-4681c68594dd\") " pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.616155 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.616114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:02.638642 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.638620 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5fp5p" Sep 09 19:56:02.651269 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651341 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-crio-socket\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-data-volume\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651437 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651481 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scx8z\" (UniqueName: \"kubernetes.io/projected/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-api-access-scx8z\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651564 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-crio-socket\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-data-volume\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.651975 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.651956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.656953 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.656907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.659319 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.659298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scx8z\" (UniqueName: \"kubernetes.io/projected/8bf7caa8-a362-4cf9-82a3-d57f577e21e0-kube-api-access-scx8z\") pod \"insights-runtime-extractor-4z8tv\" (UID: \"8bf7caa8-a362-4cf9-82a3-d57f577e21e0\") " pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.697577 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.697473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4z8tv" Sep 09 19:56:02.922621 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.922536 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5fp5p"] Sep 09 19:56:02.925994 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:02.925964 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35286300_d6c0_42e7_83f2_9c9ecec02a87.slice/crio-0ee41406ed5c38fc1a37d9fe35fb78a11b74572d248f956a2f681c4903706933 WatchSource:0}: Error finding container 0ee41406ed5c38fc1a37d9fe35fb78a11b74572d248f956a2f681c4903706933: Status 404 returned error can't find the container with id 0ee41406ed5c38fc1a37d9fe35fb78a11b74572d248f956a2f681c4903706933 Sep 09 19:56:02.929211 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.929173 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9vzms"] Sep 09 19:56:02.933465 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:02.933440 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2043cc_0dd5_48b3_b0a8_4681c68594dd.slice/crio-d09465e654044887f4e27901ccb1ecfb0c05bf0ba4b776977f4473370b059223 WatchSource:0}: Error finding container d09465e654044887f4e27901ccb1ecfb0c05bf0ba4b776977f4473370b059223: Status 404 returned error can't find the container with id d09465e654044887f4e27901ccb1ecfb0c05bf0ba4b776977f4473370b059223 Sep 09 19:56:02.988389 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:02.988359 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4z8tv"] Sep 09 19:56:02.991671 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:02.991631 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf7caa8_a362_4cf9_82a3_d57f577e21e0.slice/crio-a4537f323da995585a031c44c97fed9af39b8307f54a0c8a085cf8f51efc5e93 WatchSource:0}: Error finding container a4537f323da995585a031c44c97fed9af39b8307f54a0c8a085cf8f51efc5e93: Status 404 returned error can't find the container with id a4537f323da995585a031c44c97fed9af39b8307f54a0c8a085cf8f51efc5e93 Sep 09 19:56:03.272206 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:03.272119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z8tv" event={"ID":"8bf7caa8-a362-4cf9-82a3-d57f577e21e0","Type":"ContainerStarted","Data":"46da688f341164935ade73c6afd779126cfd03932b2725dc69cabe3d3d2672c2"} Sep 09 19:56:03.272206 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:03.272191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z8tv" event={"ID":"8bf7caa8-a362-4cf9-82a3-d57f577e21e0","Type":"ContainerStarted","Data":"a4537f323da995585a031c44c97fed9af39b8307f54a0c8a085cf8f51efc5e93"} Sep 09 19:56:03.273456 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:03.273414 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5fp5p" event={"ID":"35286300-d6c0-42e7-83f2-9c9ecec02a87","Type":"ContainerStarted","Data":"0ee41406ed5c38fc1a37d9fe35fb78a11b74572d248f956a2f681c4903706933"} Sep 09 19:56:03.274646 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:03.274621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vzms" event={"ID":"5b2043cc-0dd5-48b3-b0a8-4681c68594dd","Type":"ContainerStarted","Data":"d09465e654044887f4e27901ccb1ecfb0c05bf0ba4b776977f4473370b059223"} Sep 09 19:56:04.009924 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.009890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:04.009924 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.009920 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:04.010158 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.010080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:04.013520 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013500 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 19:56:04.013693 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013559 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\"" Sep 09 19:56:04.013809 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013785 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 19:56:04.013916 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013795 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 19:56:04.013916 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013856 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 19:56:04.013916 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:04.013879 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\"" Sep 09 19:56:05.282534 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:05.282487 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z8tv" event={"ID":"8bf7caa8-a362-4cf9-82a3-d57f577e21e0","Type":"ContainerStarted","Data":"75780430da53265e67f89f68678ebcdd56a90d3a6903d96ecfe780f0749707b9"} Sep 09 19:56:06.288029 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:06.287989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5fp5p" event={"ID":"35286300-d6c0-42e7-83f2-9c9ecec02a87","Type":"ContainerStarted","Data":"b8096d001c0fdab80f1d45278331ac5cdbd49ccc7d0bcc32fe647b63d8ec4490"} Sep 09 19:56:06.290131 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:06.290105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vzms" event={"ID":"5b2043cc-0dd5-48b3-b0a8-4681c68594dd","Type":"ContainerStarted","Data":"ec8f4a02676746abbdbd04d96905ab323a65bb09415209fe17eb8d43f206b3ba"} Sep 09 19:56:06.305451 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:06.305387 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5fp5p" podStartSLOduration=1.217551607 podStartE2EDuration="4.305366215s" totalImagesPullingTime="3.087814608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:06.304962415 +0000 UTC m=+52.823842141" watchObservedRunningTime="2026-09-09 19:56:06.305366215 +0000 UTC m=+52.824245929" Sep 09 19:56:07.295559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:07.295511 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z8tv" event={"ID":"8bf7caa8-a362-4cf9-82a3-d57f577e21e0","Type":"ContainerStarted","Data":"d560b6ccf3ee197ab68fa836f1a4fbc6882131b16c2dd1c572ce1023d3cc54f9"} Sep 09 19:56:07.297569 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:07.297537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9vzms" event={"ID":"5b2043cc-0dd5-48b3-b0a8-4681c68594dd","Type":"ContainerStarted","Data":"b840225b6bbe4fe67bc8080dcc26535ecfcae8db78de84dfd41949f0ce04e840"} Sep 09 19:56:07.297733 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:07.297718 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:07.313677 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:07.313629 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4z8tv" podStartSLOduration=2.068522345 podStartE2EDuration="5.313613229s" totalImagesPullingTime="3.245090884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:07.31244473 +0000 UTC m=+53.831324442" watchObservedRunningTime="2026-09-09 19:56:07.313613229 +0000 UTC m=+53.832492942" Sep 09 19:56:07.330494 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:07.330444 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9vzms" podStartSLOduration=2.250235344 podStartE2EDuration="5.330423895s" totalImagesPullingTime="3.080188551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:07.330422837 +0000 UTC m=+53.849302571" watchObservedRunningTime="2026-09-09 19:56:07.330423895 +0000 UTC m=+53.849303604" Sep 09 19:56:12.202129 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.202097 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd"] Sep 09 19:56:12.229861 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.229833 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd"] Sep 09 19:56:12.230007 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.229928 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:12.232662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.232635 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 19:56:12.232662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.232654 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8bkbm\"" Sep 09 19:56:12.320927 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.320889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8e6ba84-31bf-491a-b07b-5049b457811d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hpjxd\" (UID: \"d8e6ba84-31bf-491a-b07b-5049b457811d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:12.421488 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.421447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8e6ba84-31bf-491a-b07b-5049b457811d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hpjxd\" (UID: \"d8e6ba84-31bf-491a-b07b-5049b457811d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:12.423974 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.423944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8e6ba84-31bf-491a-b07b-5049b457811d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-hpjxd\" (UID: \"d8e6ba84-31bf-491a-b07b-5049b457811d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:12.539035 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.539000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:12.809016 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:12.808928 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd"] Sep 09 19:56:12.812730 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:12.812704 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e6ba84_31bf_491a_b07b_5049b457811d.slice/crio-2859a3cc57b3a0596e90c825d7c744468985f2f8cb90de1269a89475ecaf1acc WatchSource:0}: Error finding container 2859a3cc57b3a0596e90c825d7c744468985f2f8cb90de1269a89475ecaf1acc: Status 404 returned error can't find the container with id 2859a3cc57b3a0596e90c825d7c744468985f2f8cb90de1269a89475ecaf1acc Sep 09 19:56:13.223633 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.223541 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjnht" Sep 09 19:56:13.324448 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.324411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" event={"ID":"d8e6ba84-31bf-491a-b07b-5049b457811d","Type":"ContainerStarted","Data":"2859a3cc57b3a0596e90c825d7c744468985f2f8cb90de1269a89475ecaf1acc"} Sep 09 19:56:13.909481 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.909449 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:56:13.911476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.911457 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.915322 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.915294 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 19:56:13.915468 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.915368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5hng\"" Sep 09 19:56:13.915726 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.915710 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 19:56:13.915823 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.915711 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 19:56:13.916410 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.916131 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 19:56:13.916410 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.916398 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 19:56:13.916580 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.916511 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 19:56:13.916645 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.916637 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 19:56:13.922393 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.922373 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:56:13.931754 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.931922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.931922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.931922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.931922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:13.931922 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:13.931874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwfl\" (UniqueName: \"kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032170 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032184 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwfl\" (UniqueName: \"kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.032864 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.032844 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.033307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.033281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.033579 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.033557 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.037248 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.036380 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.037248 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.036568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.041023 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.041001 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwfl\" (UniqueName: \"kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl\") pod \"console-6cc7dd5665-m8tgl\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.221881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.221853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:14.333569 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.333535 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" event={"ID":"d8e6ba84-31bf-491a-b07b-5049b457811d","Type":"ContainerStarted","Data":"c3cbb0431fb8075e8366b2423b9b78f4789bac10a90a6474a51a763780bc8ff4"} Sep 09 19:56:14.334037 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.333843 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:14.338664 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.338640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" Sep 09 19:56:14.349651 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.349590 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-hpjxd" podStartSLOduration=0.996534292 podStartE2EDuration="2.349572744s" totalImagesPullingTime="1.353038452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:14.3493239 +0000 UTC m=+60.868203611" watchObservedRunningTime="2026-09-09 19:56:14.349572744 +0000 UTC m=+60.868452456" Sep 09 19:56:14.496557 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:14.496482 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:56:14.499915 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:14.499887 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c98ec18_68d4_4bf2_9868_3e22603e3a72.slice/crio-ec80542777f7a09fd181e1113dbdecd0d0c2da86672be72fbec9d0e58d9d0ebe WatchSource:0}: Error finding container ec80542777f7a09fd181e1113dbdecd0d0c2da86672be72fbec9d0e58d9d0ebe: Status 404 returned error can't find the container with id ec80542777f7a09fd181e1113dbdecd0d0c2da86672be72fbec9d0e58d9d0ebe Sep 09 19:56:15.258324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.258292 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b26p7"] Sep 09 19:56:15.260744 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.260725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.263050 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 09 19:56:15.263050 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263048 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 09 19:56:15.263280 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263049 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 19:56:15.263503 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263488 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 19:56:15.263574 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8x5wq\"" Sep 09 19:56:15.263816 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.263799 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 19:56:15.270905 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.270882 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b26p7"] Sep 09 19:56:15.339025 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.338991 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc7dd5665-m8tgl" event={"ID":"4c98ec18-68d4-4bf2-9868-3e22603e3a72","Type":"ContainerStarted","Data":"ec80542777f7a09fd181e1113dbdecd0d0c2da86672be72fbec9d0e58d9d0ebe"} Sep 09 19:56:15.340424 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.340402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.340504 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.340432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.340504 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.340456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmf74\" (UniqueName: \"kubernetes.io/projected/61899c1a-a222-421a-bf70-c85c602887d5-kube-api-access-jmf74\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.340504 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.340477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61899c1a-a222-421a-bf70-c85c602887d5-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.441350 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.441313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmf74\" (UniqueName: \"kubernetes.io/projected/61899c1a-a222-421a-bf70-c85c602887d5-kube-api-access-jmf74\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.441438 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.441359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61899c1a-a222-421a-bf70-c85c602887d5-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.441533 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.441514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.441575 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.441547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.441646 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:15.441626 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 09 19:56:15.441723 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:15.441712 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls podName:61899c1a-a222-421a-bf70-c85c602887d5 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:15.941692219 +0000 UTC m=+62.460571911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-b26p7" (UID: "61899c1a-a222-421a-bf70-c85c602887d5") : secret "prometheus-operator-tls" not found Sep 09 19:56:15.442124 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.442100 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61899c1a-a222-421a-bf70-c85c602887d5-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.444142 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.444114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.450624 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.450598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmf74\" (UniqueName: \"kubernetes.io/projected/61899c1a-a222-421a-bf70-c85c602887d5-kube-api-access-jmf74\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.944916 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.944859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:15.947408 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:15.947386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61899c1a-a222-421a-bf70-c85c602887d5-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b26p7\" (UID: \"61899c1a-a222-421a-bf70-c85c602887d5\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:16.169817 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:16.169777 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" Sep 09 19:56:16.438511 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:16.438479 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b26p7"] Sep 09 19:56:16.441775 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:16.441749 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61899c1a_a222_421a_bf70_c85c602887d5.slice/crio-c80781c4771e42148605eb58e1354221ae374ef992b1ee305fde7182d24e72a7 WatchSource:0}: Error finding container c80781c4771e42148605eb58e1354221ae374ef992b1ee305fde7182d24e72a7: Status 404 returned error can't find the container with id c80781c4771e42148605eb58e1354221ae374ef992b1ee305fde7182d24e72a7 Sep 09 19:56:17.306848 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:17.306812 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9vzms" Sep 09 19:56:17.348870 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:17.348833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" event={"ID":"61899c1a-a222-421a-bf70-c85c602887d5","Type":"ContainerStarted","Data":"c80781c4771e42148605eb58e1354221ae374ef992b1ee305fde7182d24e72a7"} Sep 09 19:56:18.354802 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:18.354765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" event={"ID":"61899c1a-a222-421a-bf70-c85c602887d5","Type":"ContainerStarted","Data":"c9108a9e4fe9523e53fc6e046d217a8c0c4f1508c2f0c847372145b8b4aa3cdf"} Sep 09 19:56:18.355188 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:18.354832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" event={"ID":"61899c1a-a222-421a-bf70-c85c602887d5","Type":"ContainerStarted","Data":"31e675eae7b696bd781a7e5d49264a6cde0b3108b55fb6e4f3ec3578af5f96da"} Sep 09 19:56:18.375216 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:18.375160 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-b26p7" podStartSLOduration=1.9752525680000002 podStartE2EDuration="3.37513902s" totalImagesPullingTime="1.399886452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:18.374569779 +0000 UTC m=+64.893449494" watchObservedRunningTime="2026-09-09 19:56:18.37513902 +0000 UTC m=+64.894018729" Sep 09 19:56:19.675993 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.675901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:19.675993 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.675972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:19.678765 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.678741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 19:56:19.678835 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.678741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 19:56:19.688965 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.688930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cd560bd-2201-4588-a031-5c4ab028f42f-original-pull-secret\") pod \"global-pull-secret-syncer-wdk27\" (UID: \"0cd560bd-2201-4588-a031-5c4ab028f42f\") " pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:19.689100 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.689031 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68bbb879-fc6e-442b-b100-0ce397467ceb-metrics-certs\") pod \"network-metrics-daemon-fq6p4\" (UID: \"68bbb879-fc6e-442b-b100-0ce397467ceb\") " pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:19.776507 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.776462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:19.779360 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.779313 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 19:56:19.789819 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.789780 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 19:56:19.800352 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.800322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkhv\" (UniqueName: \"kubernetes.io/projected/00e74531-8617-4f73-b0f6-ad325c79086f-kube-api-access-2fkhv\") pod \"network-check-target-l6hvj\" (UID: \"00e74531-8617-4f73-b0f6-ad325c79086f\") " pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:19.925214 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.925177 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\"" Sep 09 19:56:19.933163 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.933091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:19.934171 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.934152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\"" Sep 09 19:56:19.939061 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.939035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdk27" Sep 09 19:56:19.942739 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:19.942717 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fq6p4" Sep 09 19:56:20.210900 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.210817 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6hvj"] Sep 09 19:56:20.213868 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:20.213848 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e74531_8617_4f73_b0f6_ad325c79086f.slice/crio-b922630bb35acfbc53cc9cf61ab59b4f24e03a1bd077457d8899ef181da66083 WatchSource:0}: Error finding container b922630bb35acfbc53cc9cf61ab59b4f24e03a1bd077457d8899ef181da66083: Status 404 returned error can't find the container with id b922630bb35acfbc53cc9cf61ab59b4f24e03a1bd077457d8899ef181da66083 Sep 09 19:56:20.243538 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.243507 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdk27"] Sep 09 19:56:20.246961 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:20.246939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd560bd_2201_4588_a031_5c4ab028f42f.slice/crio-6c75c6f84cd0402a09881aaa0cfa845debfcf56ebcfbf12f7623f3366d67792c WatchSource:0}: Error finding container 6c75c6f84cd0402a09881aaa0cfa845debfcf56ebcfbf12f7623f3366d67792c: Status 404 returned error can't find the container with id 6c75c6f84cd0402a09881aaa0cfa845debfcf56ebcfbf12f7623f3366d67792c Sep 09 19:56:20.255508 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.255480 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fq6p4"] Sep 09 19:56:20.259262 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:20.259237 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bbb879_fc6e_442b_b100_0ce397467ceb.slice/crio-84e2df2fb341b61648be5b07a01614214dadeb07c42b1fc6b68bea5fde4c4f46 WatchSource:0}: Error finding container 84e2df2fb341b61648be5b07a01614214dadeb07c42b1fc6b68bea5fde4c4f46: Status 404 returned error can't find the container with id 84e2df2fb341b61648be5b07a01614214dadeb07c42b1fc6b68bea5fde4c4f46 Sep 09 19:56:20.362354 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.362312 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fq6p4" event={"ID":"68bbb879-fc6e-442b-b100-0ce397467ceb","Type":"ContainerStarted","Data":"84e2df2fb341b61648be5b07a01614214dadeb07c42b1fc6b68bea5fde4c4f46"} Sep 09 19:56:20.363610 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.363586 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdk27" event={"ID":"0cd560bd-2201-4588-a031-5c4ab028f42f","Type":"ContainerStarted","Data":"6c75c6f84cd0402a09881aaa0cfa845debfcf56ebcfbf12f7623f3366d67792c"} Sep 09 19:56:20.365177 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.365153 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc7dd5665-m8tgl" event={"ID":"4c98ec18-68d4-4bf2-9868-3e22603e3a72","Type":"ContainerStarted","Data":"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639"} Sep 09 19:56:20.366372 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.366346 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6hvj" event={"ID":"00e74531-8617-4f73-b0f6-ad325c79086f","Type":"ContainerStarted","Data":"b922630bb35acfbc53cc9cf61ab59b4f24e03a1bd077457d8899ef181da66083"} Sep 09 19:56:20.382833 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.382787 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cc7dd5665-m8tgl" podStartSLOduration=2.47650109 podStartE2EDuration="7.382770066s" totalImagesPullingTime="4.906268976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:20.382510151 +0000 UTC m=+66.901389864" watchObservedRunningTime="2026-09-09 19:56:20.382770066 +0000 UTC m=+66.901649772" Sep 09 19:56:20.633895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.633865 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc"] Sep 09 19:56:20.637770 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.637753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.639943 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.639921 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 19:56:20.640075 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.640028 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 19:56:20.640075 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.640065 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mz94c\"" Sep 09 19:56:20.643744 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.643723 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-tcp66"] Sep 09 19:56:20.646462 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.646442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.648869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.648850 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 19:56:20.649098 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.649075 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 19:56:20.649173 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.649136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bn9m7\"" Sep 09 19:56:20.649173 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.649141 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 19:56:20.656469 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.656446 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc"] Sep 09 19:56:20.664728 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.664702 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-tcp66"] Sep 09 19:56:20.684727 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9tn\" (UniqueName: \"kubernetes.io/projected/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-api-access-lh9tn\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.684727 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7b348e-aec4-4099-9660-e66def008548-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42hd\" (UniqueName: \"kubernetes.io/projected/0a7b348e-aec4-4099-9660-e66def008548-kube-api-access-q42hd\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.684986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.685005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d34d8f4-fd11-4824-91fc-9b7784d60775-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.685181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.685032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.694016 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.693989 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w7hv4"] Sep 09 19:56:20.696989 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.696972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.699366 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.699344 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 19:56:20.699495 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.699479 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 19:56:20.699720 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.699702 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 19:56:20.700046 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.700032 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9mskg\"" Sep 09 19:56:20.716787 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.716761 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:20.720752 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.720685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.728564 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.728538 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 19:56:20.737303 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.737272 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:20.785879 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.785844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.785899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.785947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfgq\" (UniqueName: \"kubernetes.io/projected/57a1a81e-185f-45c0-b337-947b78db271a-kube-api-access-rcfgq\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.785982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d34d8f4-fd11-4824-91fc-9b7784d60775-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-textfile\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-metrics-client-ca\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:20.786117 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-wtmp\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:20.786191 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls podName:0a7b348e-aec4-4099-9660-e66def008548 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.286168423 +0000 UTC m=+67.805048128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-df9cc" (UID: "0a7b348e-aec4-4099-9660-e66def008548") : secret "openshift-state-metrics-tls" not found Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786274 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.786398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-sys\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-root\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7d34d8f4-fd11-4824-91fc-9b7784d60775-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9tn\" (UniqueName: \"kubernetes.io/projected/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-api-access-lh9tn\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.786815 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7b348e-aec4-4099-9660-e66def008548-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzjx\" (UniqueName: \"kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.787256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.786955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42hd\" (UniqueName: \"kubernetes.io/projected/0a7b348e-aec4-4099-9660-e66def008548-kube-api-access-q42hd\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.788151 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.788041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.788525 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.788492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a7b348e-aec4-4099-9660-e66def008548-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.788923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.788899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.789676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.789652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.789808 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.789713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.796841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.796815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9tn\" (UniqueName: \"kubernetes.io/projected/7d34d8f4-fd11-4824-91fc-9b7784d60775-kube-api-access-lh9tn\") pod \"kube-state-metrics-74cb57df7-tcp66\" (UID: \"7d34d8f4-fd11-4824-91fc-9b7784d60775\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:20.797011 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.796926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42hd\" (UniqueName: \"kubernetes.io/projected/0a7b348e-aec4-4099-9660-e66def008548-kube-api-access-q42hd\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.887988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-root\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzjx\" (UniqueName: \"kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfgq\" (UniqueName: \"kubernetes.io/projected/57a1a81e-185f-45c0-b337-947b78db271a-kube-api-access-rcfgq\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-textfile\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888296 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-metrics-client-ca\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-wtmp\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-sys\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.888421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.891024 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.889138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.889208 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-root\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.889903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.890388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-wtmp\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:20.890562 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:20.890641 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls podName:57a1a81e-185f-45c0-b337-947b78db271a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.390619654 +0000 UTC m=+67.909499351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls") pod "node-exporter-w7hv4" (UID: "57a1a81e-185f-45c0-b337-947b78db271a") : secret "node-exporter-tls" not found Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.890785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-textfile\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.890858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1a81e-185f-45c0-b337-947b78db271a-sys\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.890893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-metrics-client-ca\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.891184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-accelerators-collector-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.891513 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.892071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.891555 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.893260 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.893216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.893417 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.893396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.895186 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.895151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.904349 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.904314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfgq\" (UniqueName: \"kubernetes.io/projected/57a1a81e-185f-45c0-b337-947b78db271a-kube-api-access-rcfgq\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:20.904510 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.904453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzjx\" (UniqueName: \"kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx\") pod \"console-6cdcd6d7f8-b2qtg\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:20.954760 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:20.954713 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" Sep 09 19:56:21.030546 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.030463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:21.248832 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.248754 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-tcp66"] Sep 09 19:56:21.292039 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.291996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:21.295138 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.295110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a7b348e-aec4-4099-9660-e66def008548-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df9cc\" (UID: \"0a7b348e-aec4-4099-9660-e66def008548\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:21.331704 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.331672 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:21.339866 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:21.339829 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d34d8f4_fd11_4824_91fc_9b7784d60775.slice/crio-58716b1ba91c3291f1fe914e42962bbcb82f1440d2722ef94966301b10e466d1 WatchSource:0}: Error finding container 58716b1ba91c3291f1fe914e42962bbcb82f1440d2722ef94966301b10e466d1: Status 404 returned error can't find the container with id 58716b1ba91c3291f1fe914e42962bbcb82f1440d2722ef94966301b10e466d1 Sep 09 19:56:21.341019 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:21.340997 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dff8311_75ca_4bce_9284_d76feff0e1a3.slice/crio-c0452e97dff3c0c63b58b6c8209a050df100af699274262faded4ced68096789 WatchSource:0}: Error finding container c0452e97dff3c0c63b58b6c8209a050df100af699274262faded4ced68096789: Status 404 returned error can't find the container with id c0452e97dff3c0c63b58b6c8209a050df100af699274262faded4ced68096789 Sep 09 19:56:21.371772 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.371682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" event={"ID":"7d34d8f4-fd11-4824-91fc-9b7784d60775","Type":"ContainerStarted","Data":"58716b1ba91c3291f1fe914e42962bbcb82f1440d2722ef94966301b10e466d1"} Sep 09 19:56:21.373219 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.373188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcd6d7f8-b2qtg" event={"ID":"3dff8311-75ca-4bce-9284-d76feff0e1a3","Type":"ContainerStarted","Data":"c0452e97dff3c0c63b58b6c8209a050df100af699274262faded4ced68096789"} Sep 09 19:56:21.392483 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.392441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:21.396144 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.396079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57a1a81e-185f-45c0-b337-947b78db271a-node-exporter-tls\") pod \"node-exporter-w7hv4\" (UID: \"57a1a81e-185f-45c0-b337-947b78db271a\") " pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:21.547754 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.547523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" Sep 09 19:56:21.607669 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.607635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w7hv4" Sep 09 19:56:21.629114 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:21.625943 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a1a81e_185f_45c0_b337_947b78db271a.slice/crio-d0bd4e67a8fb21670d2974ae33c60496220597c49516445739daa8e34265eba7 WatchSource:0}: Error finding container d0bd4e67a8fb21670d2974ae33c60496220597c49516445739daa8e34265eba7: Status 404 returned error can't find the container with id d0bd4e67a8fb21670d2974ae33c60496220597c49516445739daa8e34265eba7 Sep 09 19:56:21.719270 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.719194 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:56:21.723610 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.723584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.726304 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.726205 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fbk89\"" Sep 09 19:56:21.728526 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.728314 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 19:56:21.735663 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.735475 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 19:56:21.736952 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.736919 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 19:56:21.737409 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.737392 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 19:56:21.737740 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.737718 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 19:56:21.738324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.738306 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 19:56:21.739406 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.739386 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 19:56:21.745620 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.745489 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 19:56:21.745620 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.745563 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 19:56:21.763080 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.762989 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:56:21.797750 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.797923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.797923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.797923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjvs\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.797977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798127 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798127 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798205 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798205 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798317 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.798317 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.798265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.897668 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.897405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc"] Sep 09 19:56:21.903885 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.903854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.903983 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.903919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.903983 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.903964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904092 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.903998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904092 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904092 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904275 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904275 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904275 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904275 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904498 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904498 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.904498 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.904368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjvs\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.908654 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.908308 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.909121 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.908753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.909271 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.909027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.909760 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.909617 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.910941 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.910902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.915216 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.915186 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.916713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.915689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.916713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.915836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.916713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.916172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.916713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.916467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.916713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.916514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.918995 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.918950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:21.919772 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:21.919714 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjvs\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs\") pod \"alertmanager-main-0\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:22.040669 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.040405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:56:22.377770 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.377745 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:56:22.380324 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:22.380284 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8235603_e3b3_40cc_b284_24ec53be4e61.slice/crio-df871345aa4cf46341d15e4f5fd8a0ca0e8892ac4c3248e7a38f2d471bf3a8ce WatchSource:0}: Error finding container df871345aa4cf46341d15e4f5fd8a0ca0e8892ac4c3248e7a38f2d471bf3a8ce: Status 404 returned error can't find the container with id df871345aa4cf46341d15e4f5fd8a0ca0e8892ac4c3248e7a38f2d471bf3a8ce Sep 09 19:56:22.381899 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.381866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcd6d7f8-b2qtg" event={"ID":"3dff8311-75ca-4bce-9284-d76feff0e1a3","Type":"ContainerStarted","Data":"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9"} Sep 09 19:56:22.385862 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.385833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7hv4" event={"ID":"57a1a81e-185f-45c0-b337-947b78db271a","Type":"ContainerStarted","Data":"d0bd4e67a8fb21670d2974ae33c60496220597c49516445739daa8e34265eba7"} Sep 09 19:56:22.390013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.389411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fq6p4" event={"ID":"68bbb879-fc6e-442b-b100-0ce397467ceb","Type":"ContainerStarted","Data":"6423aece09fab18be9e52ace53ec2a78f942d7edd8108a93f31a2e8b3f14485a"} Sep 09 19:56:22.390013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.389469 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fq6p4" event={"ID":"68bbb879-fc6e-442b-b100-0ce397467ceb","Type":"ContainerStarted","Data":"1ccfe53a56eef7a0f9c39e4eca771e6600b0088fa89aa925b096a871f4a56ca2"} Sep 09 19:56:22.392336 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.392306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" event={"ID":"0a7b348e-aec4-4099-9660-e66def008548","Type":"ContainerStarted","Data":"08ced061f739da3017dfad2d82d81dd136851ed46c1b247b41920d97e8626ab8"} Sep 09 19:56:22.392447 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.392373 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" event={"ID":"0a7b348e-aec4-4099-9660-e66def008548","Type":"ContainerStarted","Data":"6759d2a692e21b90a5b24de13fb494f18673b98583ddbc76afb9869bdfb2853f"} Sep 09 19:56:22.392447 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.392402 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" event={"ID":"0a7b348e-aec4-4099-9660-e66def008548","Type":"ContainerStarted","Data":"c28dc3c9ab3c88887d162e3cf3ec4e6d7417cfef7d94212a12099e58401e7b7d"} Sep 09 19:56:22.435662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.435576 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fq6p4" podStartSLOduration=67.308113466 podStartE2EDuration="1m8.435515731s" totalImagesPullingTime="1.127402265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.435139488 +0000 UTC m=+68.954019270" watchObservedRunningTime="2026-09-09 19:56:22.435515731 +0000 UTC m=+68.954395445" Sep 09 19:56:22.437172 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:22.437109 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cdcd6d7f8-b2qtg" podStartSLOduration=2.437088304 podStartE2EDuration="2.437088304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.407002896 +0000 UTC m=+68.925882610" watchObservedRunningTime="2026-09-09 19:56:22.437088304 +0000 UTC m=+68.955968012" Sep 09 19:56:23.399245 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.399180 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"df871345aa4cf46341d15e4f5fd8a0ca0e8892ac4c3248e7a38f2d471bf3a8ce"} Sep 09 19:56:23.756851 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.754483 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m"] Sep 09 19:56:23.772980 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.772929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.774067 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.773774 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m"] Sep 09 19:56:23.776032 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.776001 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 19:56:23.776189 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.776165 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 19:56:23.780062 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.776452 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 19:56:23.781556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.780481 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e9fa8ig6if8tb\"" Sep 09 19:56:23.781556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.780733 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xsvc8\"" Sep 09 19:56:23.781556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.780932 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 19:56:23.781556 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.781353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 19:56:23.824478 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824685 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce16a656-448c-4447-b1a6-074254c4b4d6-metrics-client-ca\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824685 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824685 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824685 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-grpc-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824875 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824875 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4cl\" (UniqueName: \"kubernetes.io/projected/ce16a656-448c-4447-b1a6-074254c4b4d6-kube-api-access-6t4cl\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.824875 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.824802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.925998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-grpc-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4cl\" (UniqueName: \"kubernetes.io/projected/ce16a656-448c-4447-b1a6-074254c4b4d6-kube-api-access-6t4cl\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce16a656-448c-4447-b1a6-074254c4b4d6-metrics-client-ca\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.926789 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.926291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.929725 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.929659 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.930069 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.930040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce16a656-448c-4447-b1a6-074254c4b4d6-metrics-client-ca\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.933158 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.933131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.933913 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.933446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-grpc-tls\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.934022 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.933939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.936898 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.936166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.937953 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.937907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce16a656-448c-4447-b1a6-074254c4b4d6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:23.941499 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:23.941442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4cl\" (UniqueName: \"kubernetes.io/projected/ce16a656-448c-4447-b1a6-074254c4b4d6-kube-api-access-6t4cl\") pod \"thanos-querier-8dd57b9bc-rlr6m\" (UID: \"ce16a656-448c-4447-b1a6-074254c4b4d6\") " pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:24.088990 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.088942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:24.222166 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.222122 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:24.222360 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.222206 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:24.229109 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.229074 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:24.406261 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.406140 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" event={"ID":"7d34d8f4-fd11-4824-91fc-9b7784d60775","Type":"ContainerStarted","Data":"b2c12211a0db739bf2fe8229f66fd7e0ffe4b57f432a9c926c0749a8e418ec67"} Sep 09 19:56:24.410792 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:24.410761 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:56:25.141178 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.141143 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f6fc56ccd-knjvx"] Sep 09 19:56:25.143625 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.143597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.146724 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.146685 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 19:56:25.147510 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.147484 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 19:56:25.147637 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.147489 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d97kv6m0r49qd\"" Sep 09 19:56:25.148096 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.148077 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 19:56:25.148350 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.148333 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 19:56:25.148438 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.148407 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-q7fqs\"" Sep 09 19:56:25.169186 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.169153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f6fc56ccd-knjvx"] Sep 09 19:56:25.242110 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-tls\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-client-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75shv\" (UniqueName: \"kubernetes.io/projected/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-kube-api-access-75shv\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-audit-log\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242344 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-client-certs\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.242573 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.242353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-metrics-server-audit-profiles\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-tls\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-client-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75shv\" (UniqueName: \"kubernetes.io/projected/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-kube-api-access-75shv\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-audit-log\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-client-certs\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.343452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-metrics-server-audit-profiles\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.344791 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.344733 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-metrics-server-audit-profiles\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.346128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.345958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.346128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.346091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-audit-log\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.347753 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.347722 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-tls\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.347876 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.347754 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-secret-metrics-server-client-certs\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.347876 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.347822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-client-ca-bundle\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.356606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.356573 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75shv\" (UniqueName: \"kubernetes.io/projected/866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae-kube-api-access-75shv\") pod \"metrics-server-5f6fc56ccd-knjvx\" (UID: \"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae\") " pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.403369 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.403275 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc"] Sep 09 19:56:25.407833 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.407798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:25.410958 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.410680 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-smxg6\"" Sep 09 19:56:25.410958 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.410871 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 19:56:25.414454 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.414427 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc"] Sep 09 19:56:25.444310 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.444259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqhw\" (UniqueName: \"kubernetes.io/projected/3eafc3bf-71c1-43e6-9072-1cd31b92db57-kube-api-access-6cqhw\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:25.444475 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.444447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:25.457136 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.457098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:25.545610 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.545559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:25.545808 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.545654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqhw\" (UniqueName: \"kubernetes.io/projected/3eafc3bf-71c1-43e6-9072-1cd31b92db57-kube-api-access-6cqhw\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:25.545808 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:25.545731 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 09 19:56:25.545935 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:25.545812 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert podName:3eafc3bf-71c1-43e6-9072-1cd31b92db57 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:26.045791006 +0000 UTC m=+72.564670714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-6q6wc" (UID: "3eafc3bf-71c1-43e6-9072-1cd31b92db57") : secret "monitoring-plugin-cert" not found Sep 09 19:56:25.558980 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:25.558944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqhw\" (UniqueName: \"kubernetes.io/projected/3eafc3bf-71c1-43e6-9072-1cd31b92db57-kube-api-access-6cqhw\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:26.051538 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.051503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:26.054411 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.054373 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3eafc3bf-71c1-43e6-9072-1cd31b92db57-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6q6wc\" (UID: \"3eafc3bf-71c1-43e6-9072-1cd31b92db57\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:26.323814 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.323721 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:26.727214 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.727172 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:26.774593 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.774400 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:56:26.778385 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.777965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.795820 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.792374 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:56:26.862401 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.862848 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.862625 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ppx\" (UniqueName: \"kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.963938 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.963898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ppx\" (UniqueName: \"kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.963974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964099 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964394 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964823 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.964991 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.964964 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.965128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.965109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.965409 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.965386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.967369 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.967281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.967754 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.967711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:26.973144 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:26.973117 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ppx\" (UniqueName: \"kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx\") pod \"console-84c446f875-r5jl2\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:27.005725 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.005685 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f6fc56ccd-knjvx"] Sep 09 19:56:27.019763 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.019612 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:56:27.024656 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.024617 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.026917 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.026891 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 19:56:27.027152 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027127 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 19:56:27.027470 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027448 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3mpnj4ne0rsbu\"" Sep 09 19:56:27.027575 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027494 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 19:56:27.027655 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c24rk\"" Sep 09 19:56:27.027812 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027795 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 19:56:27.027894 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 19:56:27.028005 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.027988 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 19:56:27.028117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.028093 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 19:56:27.028184 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.028104 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 19:56:27.028323 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.028288 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 19:56:27.028447 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.028375 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 19:56:27.028559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.028542 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 19:56:27.029624 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.029601 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 19:56:27.029736 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.029684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 19:56:27.030056 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.030033 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 19:56:27.044726 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.044696 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:56:27.096841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.096631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.170660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168830 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.176018 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.176018 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.176018 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.176018 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.168966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttt9\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.270839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.270884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.270917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.270944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttt9\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.270991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.271035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.271066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.271098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.271141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.272117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.271169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.273931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274744 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.274744 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.274330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.279415 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.276504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.279415 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.276715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.279415 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.278254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.282169 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.281993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.284329 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.283774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.285944 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.284948 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.285944 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.285312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.289323 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.287874 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.289323 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.288723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttt9\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.291675 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.290511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.291675 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.291071 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.291675 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.291559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.295252 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.294643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.295252 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.294897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.296564 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.296097 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.296564 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.296212 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.298559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.297828 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.298559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.298489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.299151 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.299086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.299425 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.299363 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.339425 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.337891 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:27.368417 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.367773 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc"] Sep 09 19:56:27.368908 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.368873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m"] Sep 09 19:56:27.426739 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.426050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdk27" event={"ID":"0cd560bd-2201-4588-a031-5c4ab028f42f","Type":"ContainerStarted","Data":"e2e3be5b11fba037ac485b9ddbf901d264a330b7dd67486bd79e501e59192692"} Sep 09 19:56:27.431081 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.430958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" event={"ID":"7d34d8f4-fd11-4824-91fc-9b7784d60775","Type":"ContainerStarted","Data":"17a43afb7f73261da008dda705991739056a8a50f847803bf278f93c4d9dd672"} Sep 09 19:56:27.431081 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.431028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" event={"ID":"7d34d8f4-fd11-4824-91fc-9b7784d60775","Type":"ContainerStarted","Data":"68d792e7c7fa30e4bd7b686bcbf943e1dd3552147e0c6430741ec3ddf52a1538"} Sep 09 19:56:27.435881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.435844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6hvj" event={"ID":"00e74531-8617-4f73-b0f6-ad325c79086f","Type":"ContainerStarted","Data":"106a2dd5360ea516e92ff5ed22573dcff6cf0cd6224fcf20e846163a848ed119"} Sep 09 19:56:27.436755 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.436603 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:56:27.440553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.440457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" event={"ID":"0a7b348e-aec4-4099-9660-e66def008548","Type":"ContainerStarted","Data":"4bad6b25bf815a727405e1f043f2be6e96b52311d61c680fd1b66a26b86ff358"} Sep 09 19:56:27.443474 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.443441 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d" exitCode=0 Sep 09 19:56:27.443687 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.443531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d"} Sep 09 19:56:27.451046 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.450768 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" event={"ID":"3eafc3bf-71c1-43e6-9072-1cd31b92db57","Type":"ContainerStarted","Data":"6f34d603e6e7ae77cc190a7bc1fb2546fb9cba2028f67472895b56d16344d956"} Sep 09 19:56:27.460676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.456505 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"a85c2d9a7686e721d6c310b469d5453ee9f92428e89e71fd15c16d186dd1e804"} Sep 09 19:56:27.460676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.459883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7hv4" event={"ID":"57a1a81e-185f-45c0-b337-947b78db271a","Type":"ContainerStarted","Data":"00f46b2760f510a35acfdfdff4416680dec11f289eac3115e422f5d116e154bf"} Sep 09 19:56:27.463694 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.462522 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wdk27" podStartSLOduration=66.612214274 podStartE2EDuration="1m13.462500666s" totalImagesPullingTime="6.850286392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.446567043 +0000 UTC m=+73.965446756" watchObservedRunningTime="2026-09-09 19:56:27.462500666 +0000 UTC m=+73.981380380" Sep 09 19:56:27.463694 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.463443 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:56:27.465066 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.464936 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" event={"ID":"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae","Type":"ContainerStarted","Data":"63f3147b92b863044aa46972d92608dd32a3950083d1b4fedca423711f95f897"} Sep 09 19:56:27.476821 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:27.476781 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b2cbf2_d7b5_4d3a_8f67_e570d012e733.slice/crio-849aea59f772a05b7658abfa18b91233d3e5d948a08fce96d19c3ed48c69f48a WatchSource:0}: Error finding container 849aea59f772a05b7658abfa18b91233d3e5d948a08fce96d19c3ed48c69f48a: Status 404 returned error can't find the container with id 849aea59f772a05b7658abfa18b91233d3e5d948a08fce96d19c3ed48c69f48a Sep 09 19:56:27.494111 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.493657 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df9cc" podStartSLOduration=2.557691485 podStartE2EDuration="7.493640204s" totalImagesPullingTime="4.935948719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.492685791 +0000 UTC m=+74.011565502" watchObservedRunningTime="2026-09-09 19:56:27.493640204 +0000 UTC m=+74.012519917" Sep 09 19:56:27.543760 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.543704 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6hvj" podStartSLOduration=67.319570936 podStartE2EDuration="1m13.543684465s" totalImagesPullingTime="6.224113529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.542256932 +0000 UTC m=+74.061136636" watchObservedRunningTime="2026-09-09 19:56:27.543684465 +0000 UTC m=+74.062564179" Sep 09 19:56:27.543940 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.543812 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-tcp66" podStartSLOduration=5.528810888 podStartE2EDuration="7.543806247s" totalImagesPullingTime="2.014995359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:27.51763281 +0000 UTC m=+74.036512524" watchObservedRunningTime="2026-09-09 19:56:27.543806247 +0000 UTC m=+74.062685960" Sep 09 19:56:27.692889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:27.691302 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:56:27.693289 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:56:27.693254 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83335c67_42fb_448c_b8e0_8adc9415fb1b.slice/crio-4abf1e02ff8136f8b286a8f7f710dc28b5a8efaa300e76299183e425061ff76f WatchSource:0}: Error finding container 4abf1e02ff8136f8b286a8f7f710dc28b5a8efaa300e76299183e425061ff76f: Status 404 returned error can't find the container with id 4abf1e02ff8136f8b286a8f7f710dc28b5a8efaa300e76299183e425061ff76f Sep 09 19:56:28.479432 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.479396 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" exitCode=0 Sep 09 19:56:28.479917 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.479567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} Sep 09 19:56:28.480697 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.480670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"4abf1e02ff8136f8b286a8f7f710dc28b5a8efaa300e76299183e425061ff76f"} Sep 09 19:56:28.483829 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.483797 2573 generic.go:356] "Generic (PLEG): container finished" podID="57a1a81e-185f-45c0-b337-947b78db271a" containerID="00f46b2760f510a35acfdfdff4416680dec11f289eac3115e422f5d116e154bf" exitCode=0 Sep 09 19:56:28.483975 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.483951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7hv4" event={"ID":"57a1a81e-185f-45c0-b337-947b78db271a","Type":"ContainerDied","Data":"00f46b2760f510a35acfdfdff4416680dec11f289eac3115e422f5d116e154bf"} Sep 09 19:56:28.490887 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.490840 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c446f875-r5jl2" event={"ID":"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733","Type":"ContainerStarted","Data":"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d"} Sep 09 19:56:28.491171 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.490914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c446f875-r5jl2" event={"ID":"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733","Type":"ContainerStarted","Data":"849aea59f772a05b7658abfa18b91233d3e5d948a08fce96d19c3ed48c69f48a"} Sep 09 19:56:28.529801 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:28.529740 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84c446f875-r5jl2" podStartSLOduration=2.529718019 podStartE2EDuration="2.529718019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:28.527176722 +0000 UTC m=+75.046056437" watchObservedRunningTime="2026-09-09 19:56:28.529718019 +0000 UTC m=+75.048597732" Sep 09 19:56:29.499784 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:29.499683 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7hv4" event={"ID":"57a1a81e-185f-45c0-b337-947b78db271a","Type":"ContainerStarted","Data":"c87b96f2954c0b018c8c3eafdeae3d9ca20800ce964661765549404e32c77225"} Sep 09 19:56:29.502013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:29.501975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" event={"ID":"866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae","Type":"ContainerStarted","Data":"5f9cd3a0c8583e57194607f5a13fa95c747c027b121ff7a1d15f62064de0d83e"} Sep 09 19:56:29.505606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:29.505569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba"} Sep 09 19:56:29.521305 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:29.521217 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" podStartSLOduration=2.350908853 podStartE2EDuration="4.521191726s" totalImagesPullingTime="2.170282873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:29.51981724 +0000 UTC m=+76.038696956" watchObservedRunningTime="2026-09-09 19:56:29.521191726 +0000 UTC m=+76.040071437" Sep 09 19:56:30.515065 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.514964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712"} Sep 09 19:56:30.515554 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.515092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00"} Sep 09 19:56:30.515554 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.515127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac"} Sep 09 19:56:30.515554 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.515166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1"} Sep 09 19:56:30.517676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.517640 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"a2c2021268940b8cfec87607333ab26831c273477e4e6d818c4922e026f3093d"} Sep 09 19:56:30.517814 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.517726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"dd05beb71e3f93ecae11439ea4e087a9176e75e84ccccd1f7f4857cddf153e8e"} Sep 09 19:56:30.517814 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.517769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"096efc1f01da1bf457393c578e1085b8f73c674f3b08747827cddad6af4fe3ee"} Sep 09 19:56:30.520480 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.520436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w7hv4" event={"ID":"57a1a81e-185f-45c0-b337-947b78db271a","Type":"ContainerStarted","Data":"8ec920bb768bc21747cfa21189bec41d9b53345ee0abeb6d00b26ca22fbb271b"} Sep 09 19:56:30.547961 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:30.547890 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w7hv4" podStartSLOduration=4.909445962 podStartE2EDuration="10.54786471s" totalImagesPullingTime="5.499582404s" totalInitContainerRuntime="138.836344ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:30.539723637 +0000 UTC m=+77.058603352" watchObservedRunningTime="2026-09-09 19:56:30.54786471 +0000 UTC m=+77.066744422" Sep 09 19:56:31.030980 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:31.030934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:32.533773 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.533734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" event={"ID":"3eafc3bf-71c1-43e6-9072-1cd31b92db57","Type":"ContainerStarted","Data":"51f40305baba1e82747d34d3e86eb9d56e40d5f4789948e911580e1f210a0d73"} Sep 09 19:56:32.535101 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.534864 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:32.539209 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.539112 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} Sep 09 19:56:32.539209 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.539171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} Sep 09 19:56:32.541450 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.541410 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" Sep 09 19:56:32.553093 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:32.553033 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6q6wc" podStartSLOduration=2.622556799 podStartE2EDuration="7.553009425s" totalImagesPullingTime="4.930452626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:32.551040388 +0000 UTC m=+79.069920113" watchObservedRunningTime="2026-09-09 19:56:32.553009425 +0000 UTC m=+79.071889140" Sep 09 19:56:33.547632 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.547597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerStarted","Data":"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806"} Sep 09 19:56:33.551383 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.551356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} Sep 09 19:56:33.551539 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.551422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} Sep 09 19:56:33.551539 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.551460 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} Sep 09 19:56:33.551539 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.551500 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerStarted","Data":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} Sep 09 19:56:33.554559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.554531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"34edbcf88a8cffdf145b2be5b94315e18596fcb6d54809ba60f1a06a5b5d10b3"} Sep 09 19:56:33.554702 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.554598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"4534e2d068ad6612611fd141650531ec1d007964eb5065f58a5e069363b6be46"} Sep 09 19:56:33.554702 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.554628 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" event={"ID":"ce16a656-448c-4447-b1a6-074254c4b4d6","Type":"ContainerStarted","Data":"bb3e2b2aea799f693d4c227ce55fb6ab448622f0da0ef13dc37d6c39c41811f1"} Sep 09 19:56:33.574921 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.574866 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.428016199 podStartE2EDuration="12.574846136s" totalImagesPullingTime="9.120356393s" totalInitContainerRuntime="26.473544ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:33.573185602 +0000 UTC m=+80.092065353" watchObservedRunningTime="2026-09-09 19:56:33.574846136 +0000 UTC m=+80.093725859" Sep 09 19:56:33.600398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.600333 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.727509092 podStartE2EDuration="7.600310535s" totalImagesPullingTime="3.850461641s" totalInitContainerRuntime="22.339802ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:33.599211878 +0000 UTC m=+80.118091628" watchObservedRunningTime="2026-09-09 19:56:33.600310535 +0000 UTC m=+80.119190252" Sep 09 19:56:33.620592 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:33.620089 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" podStartSLOduration=5.348019241 podStartE2EDuration="10.620065344s" totalImagesPullingTime="5.272046103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:33.619276504 +0000 UTC m=+80.138156224" watchObservedRunningTime="2026-09-09 19:56:33.620065344 +0000 UTC m=+80.138945060" Sep 09 19:56:34.559607 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:34.559494 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:35.569133 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:35.569105 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8dd57b9bc-rlr6m" Sep 09 19:56:37.097657 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.097620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:37.098058 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.097687 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:37.102446 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.102424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:37.338524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.338478 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:56:37.581092 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.581058 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:56:37.632285 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:37.632252 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:56:44.217068 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:44.217032 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5fp5p_35286300-d6c0-42e7-83f2-9c9ecec02a87/serve-healthcheck-canary/0.log" Sep 09 19:56:45.457908 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:45.457873 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:45.458318 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:45.457934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:56:51.766953 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:51.766906 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cdcd6d7f8-b2qtg" podUID="3dff8311-75ca-4bce-9284-d76feff0e1a3" containerName="console" containerID="cri-o://1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9" gracePeriod=15 Sep 09 19:56:52.020406 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.020336 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:52.118823 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.118787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnzjx\" (UniqueName: \"kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119014 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.118833 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119014 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.118853 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119014 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.118886 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119014 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.118931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119014 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119002 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119302 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119034 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config\") pod \"3dff8311-75ca-4bce-9284-d76feff0e1a3\" (UID: \"3dff8311-75ca-4bce-9284-d76feff0e1a3\") " Sep 09 19:56:52.119393 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119363 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:56:52.119496 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119430 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:56:52.119627 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119452 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca" (OuterVolumeSpecName: "service-ca") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:56:52.119829 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.119800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config" (OuterVolumeSpecName: "console-config") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:56:52.121541 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.121515 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx" (OuterVolumeSpecName: "kube-api-access-tnzjx") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "kube-api-access-tnzjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:56:52.121650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.121531 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:56:52.121650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.121581 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3dff8311-75ca-4bce-9284-d76feff0e1a3" (UID: "3dff8311-75ca-4bce-9284-d76feff0e1a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:56:52.220254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220191 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnzjx\" (UniqueName: \"kubernetes.io/projected/3dff8311-75ca-4bce-9284-d76feff0e1a3-kube-api-access-tnzjx\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220241 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220251 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220261 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-service-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220269 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-oauth-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220277 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-oauth-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.220559 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.220287 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3dff8311-75ca-4bce-9284-d76feff0e1a3-console-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:56:52.641191 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.641148 2573 generic.go:356] "Generic (PLEG): container finished" podID="3dff8311-75ca-4bce-9284-d76feff0e1a3" containerID="1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9" exitCode=0 Sep 09 19:56:52.641412 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.641197 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcd6d7f8-b2qtg" Sep 09 19:56:52.641412 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.641210 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcd6d7f8-b2qtg" event={"ID":"3dff8311-75ca-4bce-9284-d76feff0e1a3","Type":"ContainerDied","Data":"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9"} Sep 09 19:56:52.641412 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.641281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcd6d7f8-b2qtg" event={"ID":"3dff8311-75ca-4bce-9284-d76feff0e1a3","Type":"ContainerDied","Data":"c0452e97dff3c0c63b58b6c8209a050df100af699274262faded4ced68096789"} Sep 09 19:56:52.641412 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.641304 2573 scope.go:117] "RemoveContainer" containerID="1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9" Sep 09 19:56:52.650410 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.650392 2573 scope.go:117] "RemoveContainer" containerID="1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9" Sep 09 19:56:52.650731 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:56:52.650712 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9\": container with ID starting with 1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9 not found: ID does not exist" containerID="1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9" Sep 09 19:56:52.650786 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.650741 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9"} err="failed to get container status \"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9\": rpc error: code = NotFound desc = could not find container \"1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9\": container with ID starting with 1a8ca030fc59aa4e7b1e8d2787d3550a622d302ec7c0b5dc8f66cfcb3e42eda9 not found: ID does not exist" Sep 09 19:56:52.663742 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.663710 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:52.667626 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:52.667594 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdcd6d7f8-b2qtg"] Sep 09 19:56:54.014617 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:54.014582 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dff8311-75ca-4bce-9284-d76feff0e1a3" path="/var/lib/kubelet/pods/3dff8311-75ca-4bce-9284-d76feff0e1a3/volumes" Sep 09 19:56:58.493495 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:56:58.493462 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6hvj" Sep 09 19:57:02.659524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:02.659474 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cc7dd5665-m8tgl" podUID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" containerName="console" containerID="cri-o://c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639" gracePeriod=15 Sep 09 19:57:02.925802 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:02.925718 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:57:03.021473 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021437 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.021650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.021650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021526 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wwfl\" (UniqueName: \"kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.021650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021603 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.021650 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021636 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.021861 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021677 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert\") pod \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\" (UID: \"4c98ec18-68d4-4bf2-9868-3e22603e3a72\") " Sep 09 19:57:03.022002 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021972 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config" (OuterVolumeSpecName: "console-config") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:03.022063 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.021978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:03.022063 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.022005 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:03.024019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.023986 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:03.024019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.023998 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl" (OuterVolumeSpecName: "kube-api-access-2wwfl") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "kube-api-access-2wwfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:57:03.024149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.024041 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c98ec18-68d4-4bf2-9868-3e22603e3a72" (UID: "4c98ec18-68d4-4bf2-9868-3e22603e3a72"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:03.122657 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122616 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwfl\" (UniqueName: \"kubernetes.io/projected/4c98ec18-68d4-4bf2-9868-3e22603e3a72-kube-api-access-2wwfl\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.122657 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122649 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-service-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.122657 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122659 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-oauth-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.122657 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122668 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.122933 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122677 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-oauth-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.122933 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.122685 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c98ec18-68d4-4bf2-9868-3e22603e3a72-console-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:03.696197 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.696155 2573 generic.go:356] "Generic (PLEG): container finished" podID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" containerID="c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639" exitCode=0 Sep 09 19:57:03.696605 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.696207 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc7dd5665-m8tgl" Sep 09 19:57:03.696605 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.696256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc7dd5665-m8tgl" event={"ID":"4c98ec18-68d4-4bf2-9868-3e22603e3a72","Type":"ContainerDied","Data":"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639"} Sep 09 19:57:03.696605 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.696310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc7dd5665-m8tgl" event={"ID":"4c98ec18-68d4-4bf2-9868-3e22603e3a72","Type":"ContainerDied","Data":"ec80542777f7a09fd181e1113dbdecd0d0c2da86672be72fbec9d0e58d9d0ebe"} Sep 09 19:57:03.696605 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.696335 2573 scope.go:117] "RemoveContainer" containerID="c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639" Sep 09 19:57:03.707746 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.707725 2573 scope.go:117] "RemoveContainer" containerID="c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639" Sep 09 19:57:03.708077 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:03.708055 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639\": container with ID starting with c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639 not found: ID does not exist" containerID="c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639" Sep 09 19:57:03.708121 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.708092 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639"} err="failed to get container status \"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639\": rpc error: code = NotFound desc = could not find container \"c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639\": container with ID starting with c2d6da176e4d54c629909237d2a99467dc436c2626734f3e1e00ad9ac8956639 not found: ID does not exist" Sep 09 19:57:03.719309 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.719275 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:57:03.723070 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:03.723040 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cc7dd5665-m8tgl"] Sep 09 19:57:04.019164 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:04.019081 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" path="/var/lib/kubelet/pods/4c98ec18-68d4-4bf2-9868-3e22603e3a72/volumes" Sep 09 19:57:05.463033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:05.462995 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:57:05.466847 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:05.466819 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f6fc56ccd-knjvx" Sep 09 19:57:27.338903 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:27.338838 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:27.359699 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:27.359663 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:27.814660 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:27.814627 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:41.034846 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.034811 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:41.035384 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035128 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="alertmanager" containerID="cri-o://200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba" gracePeriod=120 Sep 09 19:57:41.035384 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035176 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-metric" containerID="cri-o://8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712" gracePeriod=120 Sep 09 19:57:41.035384 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035289 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="config-reloader" containerID="cri-o://b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1" gracePeriod=120 Sep 09 19:57:41.035384 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035303 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy" containerID="cri-o://4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00" gracePeriod=120 Sep 09 19:57:41.035606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035218 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="prom-label-proxy" containerID="cri-o://be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806" gracePeriod=120 Sep 09 19:57:41.035606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.035245 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-web" containerID="cri-o://79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac" gracePeriod=120 Sep 09 19:57:41.856853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856822 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806" exitCode=0 Sep 09 19:57:41.856853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856847 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712" exitCode=0 Sep 09 19:57:41.856853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856853 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00" exitCode=0 Sep 09 19:57:41.856853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856860 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1" exitCode=0 Sep 09 19:57:41.856853 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856867 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba" exitCode=0 Sep 09 19:57:41.857149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806"} Sep 09 19:57:41.857149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856955 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712"} Sep 09 19:57:41.857149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.856982 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00"} Sep 09 19:57:41.857149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.857006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1"} Sep 09 19:57:41.857149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:41.857029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba"} Sep 09 19:57:42.286205 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.286177 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.388716 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388685 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388739 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388769 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388806 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388823 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388842 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388863 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.388895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389265 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.388909 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389265 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:42.389265 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389129 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 19:57:42.389434 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389277 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389434 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389434 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389377 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389579 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389438 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jjvs\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs\") pod \"c8235603-e3b3-40cc-b284-24ec53be4e61\" (UID: \"c8235603-e3b3-40cc-b284-24ec53be4e61\") " Sep 09 19:57:42.389787 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389721 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-metrics-client-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.389787 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.389743 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-main-db\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.390705 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.390672 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:42.393077 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.393049 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.393805 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.393777 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs" (OuterVolumeSpecName: "kube-api-access-8jjvs") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "kube-api-access-8jjvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:57:42.393914 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.393855 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out" (OuterVolumeSpecName: "config-out") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 19:57:42.394117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.394071 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.394277 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.394196 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.394364 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.394339 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:57:42.394427 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.394367 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.394522 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.394505 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume" (OuterVolumeSpecName: "config-volume") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.398081 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.397929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.405880 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.405850 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config" (OuterVolumeSpecName: "web-config") pod "c8235603-e3b3-40cc-b284-24ec53be4e61" (UID: "c8235603-e3b3-40cc-b284-24ec53be4e61"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:42.491168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491134 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8235603-e3b3-40cc-b284-24ec53be4e61-config-out\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491164 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-config-volume\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491174 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-main-tls\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491183 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-cluster-tls-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491192 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491202 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491210 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-web-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491219 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8235603-e3b3-40cc-b284-24ec53be4e61-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491252 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8235603-e3b3-40cc-b284-24ec53be4e61-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491260 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjvs\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-kube-api-access-8jjvs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.491421 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.491267 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8235603-e3b3-40cc-b284-24ec53be4e61-tls-assets\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:42.864876 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.864846 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerID="79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac" exitCode=0 Sep 09 19:57:42.865041 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.864905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac"} Sep 09 19:57:42.865041 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.864938 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.865041 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.864969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8235603-e3b3-40cc-b284-24ec53be4e61","Type":"ContainerDied","Data":"df871345aa4cf46341d15e4f5fd8a0ca0e8892ac4c3248e7a38f2d471bf3a8ce"} Sep 09 19:57:42.865041 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.865002 2573 scope.go:117] "RemoveContainer" containerID="be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806" Sep 09 19:57:42.873885 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.873864 2573 scope.go:117] "RemoveContainer" containerID="8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712" Sep 09 19:57:42.882120 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.882099 2573 scope.go:117] "RemoveContainer" containerID="4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00" Sep 09 19:57:42.890420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.890388 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:42.891112 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.891090 2573 scope.go:117] "RemoveContainer" containerID="79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac" Sep 09 19:57:42.893935 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.893914 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:42.899430 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.899404 2573 scope.go:117] "RemoveContainer" containerID="b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1" Sep 09 19:57:42.907518 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.907500 2573 scope.go:117] "RemoveContainer" containerID="200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba" Sep 09 19:57:42.915727 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.915705 2573 scope.go:117] "RemoveContainer" containerID="6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d" Sep 09 19:57:42.922799 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.922772 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:42.923096 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923085 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="init-config-reloader" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923100 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="init-config-reloader" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923110 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="prom-label-proxy" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923118 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="prom-label-proxy" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923129 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-metric" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923136 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-metric" Sep 09 19:57:42.923145 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923143 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923149 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923160 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dff8311-75ca-4bce-9284-d76feff0e1a3" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923165 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dff8311-75ca-4bce-9284-d76feff0e1a3" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923172 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="alertmanager" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923177 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="alertmanager" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923182 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="config-reloader" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923187 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="config-reloader" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923197 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-web" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923202 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-web" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923213 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923218 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923297 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c98ec18-68d4-4bf2-9868-3e22603e3a72" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923305 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-metric" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923312 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy-web" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923318 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dff8311-75ca-4bce-9284-d76feff0e1a3" containerName="console" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923324 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="config-reloader" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923329 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="kube-rbac-proxy" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923334 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="prom-label-proxy" Sep 09 19:57:42.923357 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.923339 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" containerName="alertmanager" Sep 09 19:57:42.924199 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.924181 2573 scope.go:117] "RemoveContainer" containerID="be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806" Sep 09 19:57:42.924594 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.924572 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806\": container with ID starting with be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806 not found: ID does not exist" containerID="be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806" Sep 09 19:57:42.924681 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.924600 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806"} err="failed to get container status \"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806\": rpc error: code = NotFound desc = could not find container \"be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806\": container with ID starting with be7aa52f9ab35ae58e69a7c43931f075b1d6dde2b1c0eed1d027a6e879b38806 not found: ID does not exist" Sep 09 19:57:42.924681 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.924626 2573 scope.go:117] "RemoveContainer" containerID="8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712" Sep 09 19:57:42.925004 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.924981 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712\": container with ID starting with 8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712 not found: ID does not exist" containerID="8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712" Sep 09 19:57:42.925070 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925011 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712"} err="failed to get container status \"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712\": rpc error: code = NotFound desc = could not find container \"8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712\": container with ID starting with 8d0cd384eb2084988435bfcb8bace9940a86922b0d611b2ed7f8b76a8cead712 not found: ID does not exist" Sep 09 19:57:42.925070 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925034 2573 scope.go:117] "RemoveContainer" containerID="4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00" Sep 09 19:57:42.925383 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.925367 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00\": container with ID starting with 4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00 not found: ID does not exist" containerID="4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00" Sep 09 19:57:42.925443 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925389 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00"} err="failed to get container status \"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00\": rpc error: code = NotFound desc = could not find container \"4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00\": container with ID starting with 4157ffc895fb15f7e1741b3399b8ab02521d0870b65f5350cb14b26b4b845b00 not found: ID does not exist" Sep 09 19:57:42.925443 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925404 2573 scope.go:117] "RemoveContainer" containerID="79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac" Sep 09 19:57:42.925699 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.925682 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac\": container with ID starting with 79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac not found: ID does not exist" containerID="79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac" Sep 09 19:57:42.925747 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925708 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac"} err="failed to get container status \"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac\": rpc error: code = NotFound desc = could not find container \"79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac\": container with ID starting with 79a61e03814e0bd894348e0e37e53883ae1ea2b81092258d03ad12536cfb75ac not found: ID does not exist" Sep 09 19:57:42.925747 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925728 2573 scope.go:117] "RemoveContainer" containerID="b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1" Sep 09 19:57:42.925992 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.925975 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1\": container with ID starting with b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1 not found: ID does not exist" containerID="b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1" Sep 09 19:57:42.926033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.925998 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1"} err="failed to get container status \"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1\": rpc error: code = NotFound desc = could not find container \"b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1\": container with ID starting with b59b15f7532ea57c4812d46ade7cb79452a7d93ca2f560d83459eea2816c04f1 not found: ID does not exist" Sep 09 19:57:42.926033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.926013 2573 scope.go:117] "RemoveContainer" containerID="200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba" Sep 09 19:57:42.926406 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.926387 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba\": container with ID starting with 200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba not found: ID does not exist" containerID="200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba" Sep 09 19:57:42.926468 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.926411 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba"} err="failed to get container status \"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba\": rpc error: code = NotFound desc = could not find container \"200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba\": container with ID starting with 200eb8bf762d933ff673116b4255ae6868cc789c10e1fc5c766be56212f690ba not found: ID does not exist" Sep 09 19:57:42.926468 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.926425 2573 scope.go:117] "RemoveContainer" containerID="6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d" Sep 09 19:57:42.926713 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:42.926690 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d\": container with ID starting with 6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d not found: ID does not exist" containerID="6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d" Sep 09 19:57:42.926804 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.926715 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d"} err="failed to get container status \"6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d\": rpc error: code = NotFound desc = could not find container \"6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d\": container with ID starting with 6bc7cd5d19d6b9fa5338d35b0117a38e7b58350efc4e4d67b05784b9a8c3446d not found: ID does not exist" Sep 09 19:57:42.927102 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.927088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.929615 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929578 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 19:57:42.929713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929646 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 19:57:42.929713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929673 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 19:57:42.929713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929593 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 19:57:42.929960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929944 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 19:57:42.930018 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.929975 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 19:57:42.930067 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.930054 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fbk89\"" Sep 09 19:57:42.930118 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.930064 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 19:57:42.930542 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.930522 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 19:57:42.935306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.935274 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 19:57:42.944919 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.944881 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:42.995908 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.995864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.995908 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.995908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.995935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.995964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-out\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-web-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjt5b\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-kube-api-access-tjt5b\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:42.996398 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:42.996214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096703 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-web-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjt5b\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-kube-api-access-tjt5b\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.096881 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.096984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.097053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.097079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-out\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097193 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.097181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.097971 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.097885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.099900 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.099867 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-out\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.100707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.100275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.100707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.100275 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-web-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.100707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.100517 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.100707 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.100654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.100944 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.100715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.101071 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.101050 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.101452 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.101427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.101900 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.101884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.102385 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.102370 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.105555 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.105529 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjt5b\" (UniqueName: \"kubernetes.io/projected/f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6-kube-api-access-tjt5b\") pod \"alertmanager-main-0\" (UID: \"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.238389 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.238296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 19:57:43.521015 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.520895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 19:57:43.870162 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.870122 2573 generic.go:356] "Generic (PLEG): container finished" podID="f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6" containerID="7dafa11bebc85d4709a95a8a0e089c9e8c73696f8527ce859de3e4d16a516be9" exitCode=0 Sep 09 19:57:43.870375 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.870210 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerDied","Data":"7dafa11bebc85d4709a95a8a0e089c9e8c73696f8527ce859de3e4d16a516be9"} Sep 09 19:57:43.870375 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:43.870308 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"cf024f381e6d536178934d2cb38bbd21b61c79b6c9488e6c08db94f2ea8077af"} Sep 09 19:57:44.014367 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.014336 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8235603-e3b3-40cc-b284-24ec53be4e61" path="/var/lib/kubelet/pods/c8235603-e3b3-40cc-b284-24ec53be4e61/volumes" Sep 09 19:57:44.879598 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879557 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"a3da8ae7ed1ddb4cc97675a5cdeeea57608da73d0de0292464557028e1851049"} Sep 09 19:57:44.880060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"aae1a7054c79553157dbd343997f023e1de57114b82404a148e3988aaf5ea388"} Sep 09 19:57:44.880060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"d61d5970e3e1f98c1c3b0b7540c24c13c6634ba36715464bdcf8db7f1ae34bb6"} Sep 09 19:57:44.880060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"e800f72c14a78c321c8432df34b91efc7f830b39a6c24f9a7a64b53ab9df4c82"} Sep 09 19:57:44.880060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"1e1b8b3ba255153b4f653ff646739c8f08d2279e761a04e465fd1bf7c198bdaa"} Sep 09 19:57:44.880060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.879789 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6","Type":"ContainerStarted","Data":"9497206b9cd28f7c25c913150f5393b653380928003da2117507fcbb76762459"} Sep 09 19:57:44.906667 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:44.906599 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.887461387 podStartE2EDuration="2.906575815s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.114428ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:44.905166023 +0000 UTC m=+151.424045748" watchObservedRunningTime="2026-09-09 19:57:44.906575815 +0000 UTC m=+151.425455528" Sep 09 19:57:45.065046 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.065004 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85885fb465-7pmvj"] Sep 09 19:57:45.067489 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.067467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.069981 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.070132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.070188 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.070376 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.070450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cdkcq\"" Sep 09 19:57:45.070548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.070481 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 19:57:45.076551 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.076529 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 19:57:45.083082 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.083047 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85885fb465-7pmvj"] Sep 09 19:57:45.115908 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.115868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116090 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.115994 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-metrics-client-ca\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116090 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116090 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116330 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116330 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-serving-certs-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116330 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116172 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-federate-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.116610 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.116582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xj79\" (UniqueName: \"kubernetes.io/projected/d3b415b3-75cf-4e0c-89d6-300ed754f65c-kube-api-access-2xj79\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217191 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217191 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-metrics-client-ca\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-serving-certs-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-federate-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.217706 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.217683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xj79\" (UniqueName: \"kubernetes.io/projected/d3b415b3-75cf-4e0c-89d6-300ed754f65c-kube-api-access-2xj79\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.218086 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.218060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-metrics-client-ca\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.218495 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.218068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-serving-certs-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.218617 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.218597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.220249 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.220188 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-federate-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.220384 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.220365 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.220505 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.220478 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.220569 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.220505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d3b415b3-75cf-4e0c-89d6-300ed754f65c-telemeter-client-tls\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.227065 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.227039 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xj79\" (UniqueName: \"kubernetes.io/projected/d3b415b3-75cf-4e0c-89d6-300ed754f65c-kube-api-access-2xj79\") pod \"telemeter-client-85885fb465-7pmvj\" (UID: \"d3b415b3-75cf-4e0c-89d6-300ed754f65c\") " pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.373737 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.373700 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374006 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="prometheus" containerID="cri-o://379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" gracePeriod=600 Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374046 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-web" containerID="cri-o://fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" gracePeriod=600 Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374040 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" gracePeriod=600 Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374051 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="thanos-sidecar" containerID="cri-o://a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" gracePeriod=600 Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374049 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy" containerID="cri-o://e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" gracePeriod=600 Sep 09 19:57:45.374306 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.374207 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="config-reloader" containerID="cri-o://800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" gracePeriod=600 Sep 09 19:57:45.380601 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.380555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" Sep 09 19:57:45.630548 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.630160 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:45.680011 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.678149 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85885fb465-7pmvj"] Sep 09 19:57:45.681882 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:57:45.681849 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b415b3_75cf_4e0c_89d6_300ed754f65c.slice/crio-1ce1fee292d277e32d24d39c4a50172931a1626105f6e20f4be3f22499a0e166 WatchSource:0}: Error finding container 1ce1fee292d277e32d24d39c4a50172931a1626105f6e20f4be3f22499a0e166: Status 404 returned error can't find the container with id 1ce1fee292d277e32d24d39c4a50172931a1626105f6e20f4be3f22499a0e166 Sep 09 19:57:45.723038 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.722995 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723038 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723043 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723076 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723201 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723269 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723372 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723400 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723433 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ttt9\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723458 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723495 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723522 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723547 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.723586 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723602 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723637 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723697 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723730 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723756 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.724013 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723801 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy\") pod \"83335c67-42fb-448c-b8e0-8adc9415fb1b\" (UID: \"83335c67-42fb-448c-b8e0-8adc9415fb1b\") " Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723540 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723575 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.723889 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.724211 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.724446 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.725813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.726124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.726423 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.727033 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.726920 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.727636 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.727487 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:57:45.728345 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.728317 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.728527 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.728489 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.728729 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.728610 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9" (OuterVolumeSpecName: "kube-api-access-5ttt9") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "kube-api-access-5ttt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:57:45.728923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.728836 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config" (OuterVolumeSpecName: "config") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.729270 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.729186 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:57:45.729723 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.729697 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out" (OuterVolumeSpecName: "config-out") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 19:57:45.730679 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.730621 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.731036 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.730993 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.731604 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.731582 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.743634 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.743554 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config" (OuterVolumeSpecName: "web-config") pod "83335c67-42fb-448c-b8e0-8adc9415fb1b" (UID: "83335c67-42fb-448c-b8e0-8adc9415fb1b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:57:45.825146 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825096 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-metrics-client-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825146 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825137 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-config-out\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825146 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825151 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825167 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttt9\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-kube-api-access-5ttt9\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825180 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825192 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-metrics-client-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825203 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825217 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825266 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-grpc-tls\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825277 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825288 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825299 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825311 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83335c67-42fb-448c-b8e0-8adc9415fb1b-tls-assets\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825322 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-web-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825333 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-kube-rbac-proxy\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825345 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825357 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825371 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83335c67-42fb-448c-b8e0-8adc9415fb1b-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825385 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.825464 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.825398 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83335c67-42fb-448c-b8e0-8adc9415fb1b-prometheus-k8s-db\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:57:45.886288 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.886255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" event={"ID":"d3b415b3-75cf-4e0c-89d6-300ed754f65c","Type":"ContainerStarted","Data":"1ce1fee292d277e32d24d39c4a50172931a1626105f6e20f4be3f22499a0e166"} Sep 09 19:57:45.889706 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889678 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" exitCode=0 Sep 09 19:57:45.889706 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889702 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" exitCode=0 Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889714 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" exitCode=0 Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889723 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" exitCode=0 Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889732 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" exitCode=0 Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889737 2573 generic.go:356] "Generic (PLEG): container finished" podID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" exitCode=0 Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889758 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} Sep 09 19:57:45.889886 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} Sep 09 19:57:45.890168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} Sep 09 19:57:45.890168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889923 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.890168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} Sep 09 19:57:45.890168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.889958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83335c67-42fb-448c-b8e0-8adc9415fb1b","Type":"ContainerDied","Data":"4abf1e02ff8136f8b286a8f7f710dc28b5a8efaa300e76299183e425061ff76f"} Sep 09 19:57:45.899070 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.899053 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.906541 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.906520 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.919491 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.915868 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.919491 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.916286 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:45.921985 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.921960 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:45.927471 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.927445 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.935019 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.934995 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.943291 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.943267 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.944832 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.944813 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:45.945154 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945142 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-thanos" Sep 09 19:57:45.945194 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945156 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-thanos" Sep 09 19:57:45.945194 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945167 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-web" Sep 09 19:57:45.945194 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945173 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-web" Sep 09 19:57:45.945194 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945183 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="thanos-sidecar" Sep 09 19:57:45.945194 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945189 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="thanos-sidecar" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945201 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="init-config-reloader" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945206 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="init-config-reloader" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945213 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="config-reloader" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945219 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="config-reloader" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945247 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945252 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945258 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="prometheus" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945263 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="prometheus" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945308 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="config-reloader" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945315 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="thanos-sidecar" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945321 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-web" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945327 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="prometheus" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945333 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy" Sep 09 19:57:45.945386 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.945337 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" containerName="kube-rbac-proxy-thanos" Sep 09 19:57:45.949468 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.949446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:45.951851 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.951829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 19:57:45.951942 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.951871 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 19:57:45.951942 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.951878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3mpnj4ne0rsbu\"" Sep 09 19:57:45.952184 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952166 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 19:57:45.952283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952256 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 19:57:45.952283 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952277 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 19:57:45.952420 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952406 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 19:57:45.952676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952470 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 19:57:45.952676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952551 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c24rk\"" Sep 09 19:57:45.952676 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952584 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 19:57:45.952821 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.952776 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 19:57:45.953097 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.953080 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 19:57:45.953173 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.953112 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 19:57:45.953713 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.953695 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 19:57:45.953819 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.953801 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.954209 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.954188 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.954321 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.954247 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.954321 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.954289 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.954684 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.954655 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.954756 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.954694 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.954756 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.954719 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.954869 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.954809 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 19:57:45.955068 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.955047 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.955123 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955077 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.955123 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955094 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.955466 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.955446 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.955565 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955475 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.955565 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955496 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.955838 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.955820 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.955923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955845 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.955923 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.955866 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.956193 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.956172 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.956276 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956200 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.956276 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956215 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.956539 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:57:45.956514 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.956621 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956545 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:45.956621 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956565 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.956957 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956932 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.957032 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.956959 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.957290 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957264 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.957290 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957289 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.957597 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957579 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.957652 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957600 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.957888 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957871 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.957958 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.957892 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.958237 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958199 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.958329 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958248 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.958439 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958417 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 19:57:45.958551 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958499 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.958551 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958521 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.958794 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958776 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:45.958841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.958811 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.959035 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959018 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.959095 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959037 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.959303 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959284 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.959359 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959304 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.959506 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959490 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.959553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959506 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.959693 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959678 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.959738 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.959692 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.962682 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.960177 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.962682 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.960203 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.963485 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.963361 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.963485 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.963392 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.964095 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964035 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:45.964208 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964097 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.964432 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964412 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.964432 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964432 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.964697 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964677 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.964796 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964711 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.964960 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964942 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.965037 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.964962 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.965281 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965258 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.965358 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965284 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.965584 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965565 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:45.965647 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965598 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.965647 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965622 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.965988 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965970 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.966074 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.965988 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.966325 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966302 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:45.966325 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966325 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.966606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966572 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.966606 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966597 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.966906 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966885 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.966906 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.966906 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.967488 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.967458 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.967488 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.967489 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.968359 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968218 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.968359 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968273 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968606 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968646 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968888 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.968907 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969114 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969132 2573 scope.go:117] "RemoveContainer" containerID="0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969395 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6"} err="failed to get container status \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": rpc error: code = NotFound desc = could not find container \"0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6\": container with ID starting with 0eb97fad390f01a51c8b6a519d38c6a2d2df37bf36f6d554ac686f9e85f773d6 not found: ID does not exist" Sep 09 19:57:45.969591 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969415 2573 scope.go:117] "RemoveContainer" containerID="e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083" Sep 09 19:57:45.969982 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969645 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083"} err="failed to get container status \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": rpc error: code = NotFound desc = could not find container \"e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083\": container with ID starting with e2ff27d1f879ede42377448ac7cb96d9ba266db9c673fdec3b46c69a5069e083 not found: ID does not exist" Sep 09 19:57:45.969982 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.969725 2573 scope.go:117] "RemoveContainer" containerID="fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2" Sep 09 19:57:45.970051 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970007 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2"} err="failed to get container status \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": rpc error: code = NotFound desc = could not find container \"fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2\": container with ID starting with fe6f82dadaa48994581bbc15835b8932a5f4cb1cadd4c4206db9686fec4c07f2 not found: ID does not exist" Sep 09 19:57:45.970051 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970025 2573 scope.go:117] "RemoveContainer" containerID="a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89" Sep 09 19:57:45.970326 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970306 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89"} err="failed to get container status \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": rpc error: code = NotFound desc = could not find container \"a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89\": container with ID starting with a56ef58dfa0d0bda19fc2beef94f636f922aa1a46982c62960e5826efb3b3e89 not found: ID does not exist" Sep 09 19:57:45.970403 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970326 2573 scope.go:117] "RemoveContainer" containerID="800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe" Sep 09 19:57:45.970580 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970563 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe"} err="failed to get container status \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": rpc error: code = NotFound desc = could not find container \"800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe\": container with ID starting with 800f3268b87d0ad67c49da3eb24ac3705853e9a9623e10808d14e27fbc8619fe not found: ID does not exist" Sep 09 19:57:45.970627 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970581 2573 scope.go:117] "RemoveContainer" containerID="379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb" Sep 09 19:57:45.970863 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970840 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb"} err="failed to get container status \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": rpc error: code = NotFound desc = could not find container \"379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb\": container with ID starting with 379022f795bd3d38779c52bff1bdd14ef11e362997dfbe54423d4370d4bac6cb not found: ID does not exist" Sep 09 19:57:45.970915 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.970866 2573 scope.go:117] "RemoveContainer" containerID="a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f" Sep 09 19:57:45.971117 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:45.971097 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f"} err="failed to get container status \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": rpc error: code = NotFound desc = could not find container \"a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f\": container with ID starting with a18344c336218c5e0157def427596d967ec33e0963c81a74d088d2577c32208f not found: ID does not exist" Sep 09 19:57:46.014501 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.014424 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83335c67-42fb-448c-b8e0-8adc9415fb1b" path="/var/lib/kubelet/pods/83335c67-42fb-448c-b8e0-8adc9415fb1b/volumes" Sep 09 19:57:46.026593 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026714 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026714 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026714 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026659 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026714 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026877 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026877 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.026877 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027004 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.026992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027060 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027112 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027112 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027307 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027377 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclv9\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-kube-api-access-bclv9\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027377 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027492 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027492 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027492 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027644 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.027644 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.027550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129015 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.128979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129015 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129286 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129479 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129524 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclv9\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-kube-api-access-bclv9\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.129889 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.129795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.130324 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.130084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.131473 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.131137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132627 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9c3ee39-db57-499f-8e33-ee7b2778106e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132707 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.132944 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.132798 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.133171 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.133287 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.133465 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133437 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.133781 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.134026 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.134026 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.133993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-config\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.134983 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.134960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.135589 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.135572 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.136159 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.136140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9c3ee39-db57-499f-8e33-ee7b2778106e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.136582 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.136564 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9c3ee39-db57-499f-8e33-ee7b2778106e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.138112 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.138096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclv9\" (UniqueName: \"kubernetes.io/projected/e9c3ee39-db57-499f-8e33-ee7b2778106e-kube-api-access-bclv9\") pod \"prometheus-k8s-0\" (UID: \"e9c3ee39-db57-499f-8e33-ee7b2778106e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.260775 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.260730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:46.549537 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.549508 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 19:57:46.550985 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:57:46.550953 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c3ee39_db57_499f_8e33_ee7b2778106e.slice/crio-dc110cb8f5ac4da519f0a3f585d73fe7b35b0a6645d88bddf1928c2555859559 WatchSource:0}: Error finding container dc110cb8f5ac4da519f0a3f585d73fe7b35b0a6645d88bddf1928c2555859559: Status 404 returned error can't find the container with id dc110cb8f5ac4da519f0a3f585d73fe7b35b0a6645d88bddf1928c2555859559 Sep 09 19:57:46.897482 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.897389 2573 generic.go:356] "Generic (PLEG): container finished" podID="e9c3ee39-db57-499f-8e33-ee7b2778106e" containerID="3e0c7aab94172a36708b8035893a19c4898b846f083bd29a6d708ee70e8bdb19" exitCode=0 Sep 09 19:57:46.897933 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.897476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerDied","Data":"3e0c7aab94172a36708b8035893a19c4898b846f083bd29a6d708ee70e8bdb19"} Sep 09 19:57:46.897933 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:46.897562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"dc110cb8f5ac4da519f0a3f585d73fe7b35b0a6645d88bddf1928c2555859559"} Sep 09 19:57:47.909834 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.909795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"edad90fc69272f400addfb16ae624f2045d4768ba214cc0da3061f460587b678"} Sep 09 19:57:47.910251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.909931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"028cfef20a9ac90e699be0e3b2a47f1231f8956f35f6d7c843425120cfe057b2"} Sep 09 19:57:47.910251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.909988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"b1b49ecfc38067a8ce52386feac9a57fc6024a9844c449038323dfcd072a43a0"} Sep 09 19:57:47.910251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.910015 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"e8e05f643266494482062ac40e58bbe006d2d8fde5bcc4d1221d49fa6e651873"} Sep 09 19:57:47.910251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.910046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"03c5a58e2407523cb8c166a19d6c223f52f5814992ef7ea43b4ca49058a7aa85"} Sep 09 19:57:47.910251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.910072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9c3ee39-db57-499f-8e33-ee7b2778106e","Type":"ContainerStarted","Data":"6ea5e93bf0a0c9ed707a30a238ae3012fd9dd04d968690da5007fc23d702041d"} Sep 09 19:57:47.912137 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.912105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" event={"ID":"d3b415b3-75cf-4e0c-89d6-300ed754f65c","Type":"ContainerStarted","Data":"ab720f2a9638c6382d04eb0e41e85c6dd29fa8c1efe9712109dfc139287f107c"} Sep 09 19:57:47.939474 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:47.939400 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9134303150000003 podStartE2EDuration="2.939377854s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.947539ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:47.938301643 +0000 UTC m=+154.457181395" watchObservedRunningTime="2026-09-09 19:57:47.939377854 +0000 UTC m=+154.458257566" Sep 09 19:57:48.919185 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:48.919143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" event={"ID":"d3b415b3-75cf-4e0c-89d6-300ed754f65c","Type":"ContainerStarted","Data":"323a74e430000f0338f5420d1d64a395e4bccdd024d3e0d8c30322e457b6e0e7"} Sep 09 19:57:48.919662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:48.919215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" event={"ID":"d3b415b3-75cf-4e0c-89d6-300ed754f65c","Type":"ContainerStarted","Data":"f721bf463e30b8be88c41b73013a89a554747d996cb2f96334557b31b305160b"} Sep 09 19:57:48.943339 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:48.943282 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85885fb465-7pmvj" podStartSLOduration=1.797761608 podStartE2EDuration="3.943259996s" totalImagesPullingTime="2.145498388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:48.942359587 +0000 UTC m=+155.461239301" watchObservedRunningTime="2026-09-09 19:57:48.943259996 +0000 UTC m=+155.462139708" Sep 09 19:57:49.044768 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.044737 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:57:49.047170 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.047147 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.065251 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.065198 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:57:49.166622 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166841 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmttx\" (UniqueName: \"kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166976 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.166976 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.166906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268269 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268269 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmttx\" (UniqueName: \"kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.268476 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.268421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.269562 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.269533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.269734 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.269611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.269796 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.269632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.269796 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.269734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.271275 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.271247 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.271387 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.271329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.276874 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.276842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmttx\" (UniqueName: \"kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx\") pod \"console-765dd6b745-7pnqb\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.356480 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.356436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:49.639777 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.639750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:57:49.642506 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:57:49.642483 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73661b1e_2acb_45c0_b95f_aa1f345de2bc.slice/crio-5576f84dd98e2f69258ebd35c58bd1497f772fc8ed160845489456c380a8c1cf WatchSource:0}: Error finding container 5576f84dd98e2f69258ebd35c58bd1497f772fc8ed160845489456c380a8c1cf: Status 404 returned error can't find the container with id 5576f84dd98e2f69258ebd35c58bd1497f772fc8ed160845489456c380a8c1cf Sep 09 19:57:49.927096 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.926990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765dd6b745-7pnqb" event={"ID":"73661b1e-2acb-45c0-b95f-aa1f345de2bc","Type":"ContainerStarted","Data":"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c"} Sep 09 19:57:49.927096 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.927052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765dd6b745-7pnqb" event={"ID":"73661b1e-2acb-45c0-b95f-aa1f345de2bc","Type":"ContainerStarted","Data":"5576f84dd98e2f69258ebd35c58bd1497f772fc8ed160845489456c380a8c1cf"} Sep 09 19:57:49.949080 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:49.949024 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765dd6b745-7pnqb" podStartSLOduration=0.949006835 podStartE2EDuration="949.006835ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:49.94594889 +0000 UTC m=+156.464828617" watchObservedRunningTime="2026-09-09 19:57:49.949006835 +0000 UTC m=+156.467886590" Sep 09 19:57:51.261181 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:51.261141 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:57:59.356740 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:59.356691 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:59.357149 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:59.356768 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:59.361512 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:59.361487 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:57:59.974662 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:57:59.974635 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:58:00.016254 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:00.016206 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:58:25.038424 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.038363 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84c446f875-r5jl2" podUID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" containerName="console" containerID="cri-o://511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d" gracePeriod=15 Sep 09 19:58:25.285089 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.285059 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:58:25.379799 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379701 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4ppx\" (UniqueName: \"kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.379799 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379765 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.379799 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379791 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.380089 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379808 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.380089 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379825 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.380089 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379863 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.380089 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.379982 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle\") pod \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\" (UID: \"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733\") " Sep 09 19:58:25.380326 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380185 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:58:25.380326 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380263 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config" (OuterVolumeSpecName: "console-config") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:58:25.380437 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380372 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.380437 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380389 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-service-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.380517 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380462 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:58:25.380561 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.380542 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:58:25.382505 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.382477 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:58:25.382600 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.382562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:58:25.382666 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.382646 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx" (OuterVolumeSpecName: "kube-api-access-v4ppx") pod "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" (UID: "b2b2cbf2-d7b5-4d3a-8f67-e570d012e733"). InnerVolumeSpecName "kube-api-access-v4ppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:58:25.481722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.481675 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.481722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.481707 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4ppx\" (UniqueName: \"kubernetes.io/projected/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-kube-api-access-v4ppx\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.481722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.481717 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-oauth-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.481722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.481726 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-oauth-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:25.481722 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:25.481735 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733-console-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:58:26.077888 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.077855 2573 generic.go:356] "Generic (PLEG): container finished" podID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" containerID="511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d" exitCode=0 Sep 09 19:58:26.078364 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.077904 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84c446f875-r5jl2" Sep 09 19:58:26.078364 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.077917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c446f875-r5jl2" event={"ID":"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733","Type":"ContainerDied","Data":"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d"} Sep 09 19:58:26.078364 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.077969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84c446f875-r5jl2" event={"ID":"b2b2cbf2-d7b5-4d3a-8f67-e570d012e733","Type":"ContainerDied","Data":"849aea59f772a05b7658abfa18b91233d3e5d948a08fce96d19c3ed48c69f48a"} Sep 09 19:58:26.078364 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.078002 2573 scope.go:117] "RemoveContainer" containerID="511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d" Sep 09 19:58:26.086528 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.086508 2573 scope.go:117] "RemoveContainer" containerID="511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d" Sep 09 19:58:26.086834 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:58:26.086812 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d\": container with ID starting with 511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d not found: ID does not exist" containerID="511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d" Sep 09 19:58:26.086895 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.086843 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d"} err="failed to get container status \"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d\": rpc error: code = NotFound desc = could not find container \"511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d\": container with ID starting with 511e3fadf55655ecb9706eab5b262d46d31912a2c4f1ee8490a692a34dbffc8d not found: ID does not exist" Sep 09 19:58:26.094097 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.094069 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:58:26.098633 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:26.098608 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84c446f875-r5jl2"] Sep 09 19:58:28.013442 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:28.013411 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" path="/var/lib/kubelet/pods/b2b2cbf2-d7b5-4d3a-8f67-e570d012e733/volumes" Sep 09 19:58:46.261394 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:46.261351 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:58:46.277934 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:46.277907 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:58:47.183006 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:47.182976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 19:58:56.867848 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.867814 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 19:58:56.868267 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.868199 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" containerName="console" Sep 09 19:58:56.868267 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.868212 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" containerName="console" Sep 09 19:58:56.868345 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.868287 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b2cbf2-d7b5-4d3a-8f67-e570d012e733" containerName="console" Sep 09 19:58:56.872475 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.872451 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.884256 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.884208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 19:58:56.963740 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.963740 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.963969 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.963969 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963833 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.963969 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkk6\" (UniqueName: \"kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.963969 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:56.964111 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:56.963972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064632 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064632 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064910 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064910 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064910 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064910 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.064910 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.064786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkk6\" (UniqueName: \"kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.065633 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.065604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.065733 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.065604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.065733 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.065660 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.065733 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.065687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.067503 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.067474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.067595 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.067563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.073474 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.073449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkk6\" (UniqueName: \"kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6\") pod \"console-d9b4c5454-6shw7\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.181598 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.181495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:58:57.482702 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:57.482677 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 19:58:57.485592 ip-10-0-141-196 kubenswrapper[2573]: W0909 19:58:57.485560 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb8a493_320f_4385_aecf_3cea55688a63.slice/crio-4111c980223f2fcbdddde8eadaeb90d8ce2fde734f946a51c82532c8e5d551a6 WatchSource:0}: Error finding container 4111c980223f2fcbdddde8eadaeb90d8ce2fde734f946a51c82532c8e5d551a6: Status 404 returned error can't find the container with id 4111c980223f2fcbdddde8eadaeb90d8ce2fde734f946a51c82532c8e5d551a6 Sep 09 19:58:58.212391 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:58.212355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9b4c5454-6shw7" event={"ID":"7bb8a493-320f-4385-aecf-3cea55688a63","Type":"ContainerStarted","Data":"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07"} Sep 09 19:58:58.212771 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:58.212409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9b4c5454-6shw7" event={"ID":"7bb8a493-320f-4385-aecf-3cea55688a63","Type":"ContainerStarted","Data":"4111c980223f2fcbdddde8eadaeb90d8ce2fde734f946a51c82532c8e5d551a6"} Sep 09 19:58:58.231605 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:58:58.231538 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9b4c5454-6shw7" podStartSLOduration=2.231515557 podStartE2EDuration="2.231515557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:58.230927159 +0000 UTC m=+224.749806883" watchObservedRunningTime="2026-09-09 19:58:58.231515557 +0000 UTC m=+224.750395271" Sep 09 19:59:07.181753 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:07.181714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:59:07.182126 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:07.181785 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:59:07.187553 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:07.187525 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:59:07.255274 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:07.255248 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 19:59:07.300276 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:07.300217 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:59:32.323523 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.323453 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765dd6b745-7pnqb" podUID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" containerName="console" containerID="cri-o://8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c" gracePeriod=15 Sep 09 19:59:32.570621 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.570595 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:59:32.667148 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667037 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmttx\" (UniqueName: \"kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667148 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667087 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667148 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667108 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667148 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667134 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667522 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667166 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667522 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667284 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667522 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667315 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert\") pod \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\" (UID: \"73661b1e-2acb-45c0-b95f-aa1f345de2bc\") " Sep 09 19:59:32.667833 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667797 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config" (OuterVolumeSpecName: "console-config") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:59:32.668128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:59:32.668128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667914 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:59:32.668128 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.667926 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 19:59:32.669670 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.669640 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:59:32.669670 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.669654 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 19:59:32.669670 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.669656 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx" (OuterVolumeSpecName: "kube-api-access-pmttx") pod "73661b1e-2acb-45c0-b95f-aa1f345de2bc" (UID: "73661b1e-2acb-45c0-b95f-aa1f345de2bc"). InnerVolumeSpecName "kube-api-access-pmttx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 19:59:32.768831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768778 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-service-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.768831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768823 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-oauth-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.768831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768833 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmttx\" (UniqueName: \"kubernetes.io/projected/73661b1e-2acb-45c0-b95f-aa1f345de2bc-kube-api-access-pmttx\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.768831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768842 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.768831 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768851 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-oauth-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.769168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768863 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:32.769168 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:32.768871 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73661b1e-2acb-45c0-b95f-aa1f345de2bc-console-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 19:59:33.360159 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.360118 2573 generic.go:356] "Generic (PLEG): container finished" podID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" containerID="8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c" exitCode=0 Sep 09 19:59:33.360596 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.360171 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765dd6b745-7pnqb" Sep 09 19:59:33.360596 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.360208 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765dd6b745-7pnqb" event={"ID":"73661b1e-2acb-45c0-b95f-aa1f345de2bc","Type":"ContainerDied","Data":"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c"} Sep 09 19:59:33.360596 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.360287 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765dd6b745-7pnqb" event={"ID":"73661b1e-2acb-45c0-b95f-aa1f345de2bc","Type":"ContainerDied","Data":"5576f84dd98e2f69258ebd35c58bd1497f772fc8ed160845489456c380a8c1cf"} Sep 09 19:59:33.360596 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.360307 2573 scope.go:117] "RemoveContainer" containerID="8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c" Sep 09 19:59:33.373366 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.373344 2573 scope.go:117] "RemoveContainer" containerID="8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c" Sep 09 19:59:33.373724 ip-10-0-141-196 kubenswrapper[2573]: E0909 19:59:33.373696 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c\": container with ID starting with 8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c not found: ID does not exist" containerID="8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c" Sep 09 19:59:33.373823 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.373731 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c"} err="failed to get container status \"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c\": rpc error: code = NotFound desc = could not find container \"8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c\": container with ID starting with 8b9ba93658f8e373839a73634c5e9f14dfc24494fee0f79c17aa78d4636a0e3c not found: ID does not exist" Sep 09 19:59:33.385574 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.385545 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:59:33.389854 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:33.389826 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765dd6b745-7pnqb"] Sep 09 19:59:34.014599 ip-10-0-141-196 kubenswrapper[2573]: I0909 19:59:34.014565 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" path="/var/lib/kubelet/pods/73661b1e-2acb-45c0-b95f-aa1f345de2bc/volumes" Sep 09 20:00:13.903057 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:00:13.903030 2573 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 20:01:08.944124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.944044 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-n9vd8"] Sep 09 20:01:08.944564 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.944419 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" containerName="console" Sep 09 20:01:08.944564 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.944430 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" containerName="console" Sep 09 20:01:08.944564 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.944480 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="73661b1e-2acb-45c0-b95f-aa1f345de2bc" containerName="console" Sep 09 20:01:08.947025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.947009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:08.949395 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.949364 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 09 20:01:08.949532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.949421 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zbjxt\"" Sep 09 20:01:08.950119 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.950104 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 09 20:01:08.950251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.950236 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 09 20:01:08.950251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.950246 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 09 20:01:08.956929 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.956908 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-n9vd8"] Sep 09 20:01:08.992026 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.991989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-certificates\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:08.992026 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:08.992028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhvn\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-kube-api-access-cxhvn\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.093539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.093502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-certificates\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.093539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.093542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhvn\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-kube-api-access-cxhvn\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.096373 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.096348 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-certificates\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.102545 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.102504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhvn\" (UniqueName: \"kubernetes.io/projected/3288a845-3fe6-405e-ae9d-467ecadaea4b-kube-api-access-cxhvn\") pod \"keda-admission-6c4d864b86-n9vd8\" (UID: \"3288a845-3fe6-405e-ae9d-467ecadaea4b\") " pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.258606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.258518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:09.541034 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.541008 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-n9vd8"] Sep 09 20:01:09.546508 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.546484 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:01:09.756698 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:09.756667 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" event={"ID":"3288a845-3fe6-405e-ae9d-467ecadaea4b","Type":"ContainerStarted","Data":"88cfa725b05d8266ce0d5feced6479f0883952f6d0ec861da690662ebc3a53c8"} Sep 09 20:01:12.772631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:12.772538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" event={"ID":"3288a845-3fe6-405e-ae9d-467ecadaea4b","Type":"ContainerStarted","Data":"ea61dd8a1336217f667eac0fa21688657f9475db79a60e2d88948da5676fefb4"} Sep 09 20:01:12.773069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:12.772667 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:12.791031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:12.790983 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" podStartSLOduration=2.039434016 podStartE2EDuration="4.790964778s" totalImagesPullingTime="2.751530762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:12.789098755 +0000 UTC m=+359.307978480" watchObservedRunningTime="2026-09-09 20:01:12.790964778 +0000 UTC m=+359.309844491" Sep 09 20:01:22.444701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.444664 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d9d87549-9l4s2"] Sep 09 20:01:22.448676 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.448651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.464925 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.464893 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d9d87549-9l4s2"] Sep 09 20:01:22.624088 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84cc\" (UniqueName: \"kubernetes.io/projected/780115ae-a50b-4cf7-9825-4de1f789abf0-kube-api-access-k84cc\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-console-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-oauth-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-oauth-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-service-ca\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.624513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.624396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-trusted-ca-bundle\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725515 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725515 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-oauth-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725738 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-service-ca\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725738 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-trusted-ca-bundle\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725738 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84cc\" (UniqueName: \"kubernetes.io/projected/780115ae-a50b-4cf7-9825-4de1f789abf0-kube-api-access-k84cc\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725738 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725733 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-console-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.725932 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.725754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-oauth-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.726334 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.726312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-service-ca\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.726704 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.726615 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-console-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.726704 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.726639 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-trusted-ca-bundle\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.726810 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.726786 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/780115ae-a50b-4cf7-9825-4de1f789abf0-oauth-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.728162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.728139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-serving-cert\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.728162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.728156 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/780115ae-a50b-4cf7-9825-4de1f789abf0-console-oauth-config\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.734847 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.734826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84cc\" (UniqueName: \"kubernetes.io/projected/780115ae-a50b-4cf7-9825-4de1f789abf0-kube-api-access-k84cc\") pod \"console-66d9d87549-9l4s2\" (UID: \"780115ae-a50b-4cf7-9825-4de1f789abf0\") " pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:22.757809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:22.757768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:23.034717 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:23.034629 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d9d87549-9l4s2"] Sep 09 20:01:23.038699 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:01:23.038671 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780115ae_a50b_4cf7_9825_4de1f789abf0.slice/crio-ba66ccff675aa7ae2066cd5e3a80fa3001d0036160a5319d30387341a16ca8ef WatchSource:0}: Error finding container ba66ccff675aa7ae2066cd5e3a80fa3001d0036160a5319d30387341a16ca8ef: Status 404 returned error can't find the container with id ba66ccff675aa7ae2066cd5e3a80fa3001d0036160a5319d30387341a16ca8ef Sep 09 20:01:23.820418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:23.820384 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9d87549-9l4s2" event={"ID":"780115ae-a50b-4cf7-9825-4de1f789abf0","Type":"ContainerStarted","Data":"a821d4eeb552e5b3bf2f362e1fa154048c3dd946158af974a62918792520f780"} Sep 09 20:01:23.820785 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:23.820429 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9d87549-9l4s2" event={"ID":"780115ae-a50b-4cf7-9825-4de1f789abf0","Type":"ContainerStarted","Data":"ba66ccff675aa7ae2066cd5e3a80fa3001d0036160a5319d30387341a16ca8ef"} Sep 09 20:01:23.840866 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:23.840807 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d9d87549-9l4s2" podStartSLOduration=1.840786442 podStartE2EDuration="1.840786442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:23.84004536 +0000 UTC m=+370.358925074" watchObservedRunningTime="2026-09-09 20:01:23.840786442 +0000 UTC m=+370.359666157" Sep 09 20:01:32.758611 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:32.758566 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:32.759098 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:32.758836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:32.763675 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:32.763655 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:32.861598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:32.861572 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d9d87549-9l4s2" Sep 09 20:01:32.912952 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:32.912925 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 20:01:33.778928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:33.778897 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-n9vd8" Sep 09 20:01:57.934795 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:57.934720 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d9b4c5454-6shw7" podUID="7bb8a493-320f-4385-aecf-3cea55688a63" containerName="console" containerID="cri-o://752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07" gracePeriod=15 Sep 09 20:01:58.174317 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.174289 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 20:01:58.244167 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244083 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbkk6\" (UniqueName: \"kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244167 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244140 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244186 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244218 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244262 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244311 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244349 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert\") pod \"7bb8a493-320f-4385-aecf-3cea55688a63\" (UID: \"7bb8a493-320f-4385-aecf-3cea55688a63\") " Sep 09 20:01:58.244682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244632 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config" (OuterVolumeSpecName: "console-config") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:01:58.244738 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244686 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:01:58.244985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244921 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:01:58.244985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.244744 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca" (OuterVolumeSpecName: "service-ca") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:01:58.246502 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.246481 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:01:58.246611 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.246502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:01:58.246611 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.246536 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6" (OuterVolumeSpecName: "kube-api-access-dbkk6") pod "7bb8a493-320f-4385-aecf-3cea55688a63" (UID: "7bb8a493-320f-4385-aecf-3cea55688a63"). InnerVolumeSpecName "kube-api-access-dbkk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:01:58.345863 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345821 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-oauth-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.345863 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345857 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-trusted-ca-bundle\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.345863 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345866 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-console-config\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.345863 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345874 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-service-ca\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.346118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345882 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8a493-320f-4385-aecf-3cea55688a63-oauth-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.346118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345891 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkk6\" (UniqueName: \"kubernetes.io/projected/7bb8a493-320f-4385-aecf-3cea55688a63-kube-api-access-dbkk6\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.346118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.345899 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb8a493-320f-4385-aecf-3cea55688a63-console-serving-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:01:58.965351 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.965310 2573 generic.go:356] "Generic (PLEG): container finished" podID="7bb8a493-320f-4385-aecf-3cea55688a63" containerID="752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07" exitCode=0 Sep 09 20:01:58.965770 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.965361 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9b4c5454-6shw7" event={"ID":"7bb8a493-320f-4385-aecf-3cea55688a63","Type":"ContainerDied","Data":"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07"} Sep 09 20:01:58.965770 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.965369 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9b4c5454-6shw7" Sep 09 20:01:58.965770 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.965443 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9b4c5454-6shw7" event={"ID":"7bb8a493-320f-4385-aecf-3cea55688a63","Type":"ContainerDied","Data":"4111c980223f2fcbdddde8eadaeb90d8ce2fde734f946a51c82532c8e5d551a6"} Sep 09 20:01:58.965770 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.965470 2573 scope.go:117] "RemoveContainer" containerID="752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07" Sep 09 20:01:58.974762 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.974745 2573 scope.go:117] "RemoveContainer" containerID="752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07" Sep 09 20:01:58.975083 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:01:58.975063 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07\": container with ID starting with 752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07 not found: ID does not exist" containerID="752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07" Sep 09 20:01:58.975132 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.975093 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07"} err="failed to get container status \"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07\": rpc error: code = NotFound desc = could not find container \"752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07\": container with ID starting with 752e1c4adcb264fc5fd883760ef6d474fa9108e2160e19ed1a500e38fdb5bd07 not found: ID does not exist" Sep 09 20:01:58.987295 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.987267 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 20:01:58.991370 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:01:58.991349 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d9b4c5454-6shw7"] Sep 09 20:02:00.013854 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:00.013826 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb8a493-320f-4385-aecf-3cea55688a63" path="/var/lib/kubelet/pods/7bb8a493-320f-4385-aecf-3cea55688a63/volumes" Sep 09 20:02:20.236742 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.236702 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lg8wk"] Sep 09 20:02:20.237258 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.237055 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb8a493-320f-4385-aecf-3cea55688a63" containerName="console" Sep 09 20:02:20.237258 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.237066 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb8a493-320f-4385-aecf-3cea55688a63" containerName="console" Sep 09 20:02:20.237258 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.237121 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb8a493-320f-4385-aecf-3cea55688a63" containerName="console" Sep 09 20:02:20.241483 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.241462 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.244042 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.243998 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 20:02:20.244776 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.244755 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8jr4j\"" Sep 09 20:02:20.244851 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.244775 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 20:02:20.257695 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.257665 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lg8wk"] Sep 09 20:02:20.342473 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.342427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.342662 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.342555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24vh\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-kube-api-access-w24vh\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.443011 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.442974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24vh\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-kube-api-access-w24vh\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.443153 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.443033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.471332 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.471296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24vh\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-kube-api-access-w24vh\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.485689 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.485622 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44286963-a413-4a91-a1a1-7639483b3841-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lg8wk\" (UID: \"44286963-a413-4a91-a1a1-7639483b3841\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.563412 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.563374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:20.862245 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:20.862198 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lg8wk"] Sep 09 20:02:20.864622 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:02:20.864588 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44286963_a413_4a91_a1a1_7639483b3841.slice/crio-2ec3710a91a45d489dbf96f6d1140b54dc16830c220d9bdabeaf511b900a23c1 WatchSource:0}: Error finding container 2ec3710a91a45d489dbf96f6d1140b54dc16830c220d9bdabeaf511b900a23c1: Status 404 returned error can't find the container with id 2ec3710a91a45d489dbf96f6d1140b54dc16830c220d9bdabeaf511b900a23c1 Sep 09 20:02:21.056801 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:21.056764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" event={"ID":"44286963-a413-4a91-a1a1-7639483b3841","Type":"ContainerStarted","Data":"2ec3710a91a45d489dbf96f6d1140b54dc16830c220d9bdabeaf511b900a23c1"} Sep 09 20:02:25.078645 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:25.078607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" event={"ID":"44286963-a413-4a91-a1a1-7639483b3841","Type":"ContainerStarted","Data":"23a1bc08bd74c82926a27602e91e597d44c8e3290ddcd44c9cf61947f2d85a36"} Sep 09 20:02:25.079102 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:25.078710 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:25.098527 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:25.098463 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" podStartSLOduration=1.5757496720000002 podStartE2EDuration="5.098444126s" totalImagesPullingTime="3.522694454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:25.097038134 +0000 UTC m=+431.615917848" watchObservedRunningTime="2026-09-09 20:02:25.098444126 +0000 UTC m=+431.617323839" Sep 09 20:02:31.084673 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:31.084643 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lg8wk" Sep 09 20:02:35.446059 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.445974 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6"] Sep 09 20:02:35.449199 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.449181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.451697 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.451672 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:02:35.451833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.451672 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 20:02:35.452414 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.452399 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-97792\"" Sep 09 20:02:35.458048 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.458021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6"] Sep 09 20:02:35.575260 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.575205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3727c8d8-80cc-4929-8dee-29c70e470c33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.575483 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.575312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkdn5\" (UniqueName: \"kubernetes.io/projected/3727c8d8-80cc-4929-8dee-29c70e470c33-kube-api-access-rkdn5\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.676464 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.676420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3727c8d8-80cc-4929-8dee-29c70e470c33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.676657 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.676475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkdn5\" (UniqueName: \"kubernetes.io/projected/3727c8d8-80cc-4929-8dee-29c70e470c33-kube-api-access-rkdn5\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.676788 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.676765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3727c8d8-80cc-4929-8dee-29c70e470c33-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.685736 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.685711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkdn5\" (UniqueName: \"kubernetes.io/projected/3727c8d8-80cc-4929-8dee-29c70e470c33-kube-api-access-rkdn5\") pod \"openshift-lws-operator-fd8ccff4c-8zxk6\" (UID: \"3727c8d8-80cc-4929-8dee-29c70e470c33\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:35.759641 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:35.759536 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" Sep 09 20:02:36.037664 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:36.037638 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6"] Sep 09 20:02:36.040399 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:02:36.040368 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3727c8d8_80cc_4929_8dee_29c70e470c33.slice/crio-ad8ca696035447b7870b6785fb42ab735ddb7f663e4a747c6639a700d0efa040 WatchSource:0}: Error finding container ad8ca696035447b7870b6785fb42ab735ddb7f663e4a747c6639a700d0efa040: Status 404 returned error can't find the container with id ad8ca696035447b7870b6785fb42ab735ddb7f663e4a747c6639a700d0efa040 Sep 09 20:02:36.124812 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:36.124779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" event={"ID":"3727c8d8-80cc-4929-8dee-29c70e470c33","Type":"ContainerStarted","Data":"ad8ca696035447b7870b6785fb42ab735ddb7f663e4a747c6639a700d0efa040"} Sep 09 20:02:39.140018 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:39.139975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" event={"ID":"3727c8d8-80cc-4929-8dee-29c70e470c33","Type":"ContainerStarted","Data":"9175f97719fe248f012a0419c011863ccecc45458191419daf6659986da9ae6a"} Sep 09 20:02:39.162911 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:02:39.162861 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zxk6" podStartSLOduration=1.231983716 podStartE2EDuration="4.162845304s" totalImagesPullingTime="2.930861588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:39.160326149 +0000 UTC m=+445.679205864" watchObservedRunningTime="2026-09-09 20:02:39.162845304 +0000 UTC m=+445.681725016" Sep 09 20:03:03.385999 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.385965 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc"] Sep 09 20:03:03.392132 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.392113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.398001 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.397969 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 20:03:03.398176 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.397999 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-f48g9\"" Sep 09 20:03:03.398176 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.397983 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 20:03:03.398352 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.398169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 20:03:03.412766 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.412737 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc"] Sep 09 20:03:03.428785 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.428751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-manager-config\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.428938 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.428794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.428938 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.428867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrxm\" (UniqueName: \"kubernetes.io/projected/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-kube-api-access-jbrxm\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.428938 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.428930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.530193 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.530152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-manager-config\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.530193 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.530195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.530418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.530272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrxm\" (UniqueName: \"kubernetes.io/projected/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-kube-api-access-jbrxm\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.530418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.530340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.530948 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.530930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-manager-config\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.533014 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.532984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.533123 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.532990 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.539692 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.539671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrxm\" (UniqueName: \"kubernetes.io/projected/9c9d333b-9f0f-46bc-8dac-0659d9dafd91-kube-api-access-jbrxm\") pod \"lws-controller-manager-7ff45979b5-pswbc\" (UID: \"9c9d333b-9f0f-46bc-8dac-0659d9dafd91\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:03.701949 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:03.701857 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:04.016822 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:04.016796 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc"] Sep 09 20:03:04.019401 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:03:04.019367 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9d333b_9f0f_46bc_8dac_0659d9dafd91.slice/crio-8f0f1047163f998a2714c638be7708c4d0473ddcb534758c64dba2c3debfac5d WatchSource:0}: Error finding container 8f0f1047163f998a2714c638be7708c4d0473ddcb534758c64dba2c3debfac5d: Status 404 returned error can't find the container with id 8f0f1047163f998a2714c638be7708c4d0473ddcb534758c64dba2c3debfac5d Sep 09 20:03:04.254946 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:04.254905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" event={"ID":"9c9d333b-9f0f-46bc-8dac-0659d9dafd91","Type":"ContainerStarted","Data":"8f0f1047163f998a2714c638be7708c4d0473ddcb534758c64dba2c3debfac5d"} Sep 09 20:03:06.268487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:06.268446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" event={"ID":"9c9d333b-9f0f-46bc-8dac-0659d9dafd91","Type":"ContainerStarted","Data":"34506cecef9ec967d0e97329accb708bc70eef2f6aa9abf4c164bfd1672023b8"} Sep 09 20:03:06.268890 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:06.268508 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:06.328677 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:06.328617 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" podStartSLOduration=1.18918791 podStartE2EDuration="3.328597474s" totalImagesPullingTime="2.139409564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:06.321327682 +0000 UTC m=+472.840207397" watchObservedRunningTime="2026-09-09 20:03:06.328597474 +0000 UTC m=+472.847477186" Sep 09 20:03:17.275839 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:17.275799 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-pswbc" Sep 09 20:03:28.883179 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.883144 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx"] Sep 09 20:03:28.885620 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.885602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:28.888308 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.888281 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nd7lp\"" Sep 09 20:03:28.888402 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.888356 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 20:03:28.889035 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.889018 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 20:03:28.889263 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.889247 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 20:03:28.904468 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.904440 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx"] Sep 09 20:03:28.967795 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:28.967757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47477\" (UniqueName: \"kubernetes.io/projected/98f942f6-35a2-42df-97fe-698ccf44bc2a-kube-api-access-47477\") pod \"dns-operator-controller-manager-65dc8f76f5-5rnjx\" (UID: \"98f942f6-35a2-42df-97fe-698ccf44bc2a\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:29.068942 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:29.068909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47477\" (UniqueName: \"kubernetes.io/projected/98f942f6-35a2-42df-97fe-698ccf44bc2a-kube-api-access-47477\") pod \"dns-operator-controller-manager-65dc8f76f5-5rnjx\" (UID: \"98f942f6-35a2-42df-97fe-698ccf44bc2a\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:29.081205 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:29.081175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47477\" (UniqueName: \"kubernetes.io/projected/98f942f6-35a2-42df-97fe-698ccf44bc2a-kube-api-access-47477\") pod \"dns-operator-controller-manager-65dc8f76f5-5rnjx\" (UID: \"98f942f6-35a2-42df-97fe-698ccf44bc2a\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:29.196948 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:29.196867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:29.473281 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:29.473143 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx"] Sep 09 20:03:29.476166 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:03:29.476138 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f942f6_35a2_42df_97fe_698ccf44bc2a.slice/crio-68285cd42beb69e36b2909d32440b9e28e68e8cd105a90198fb225921eec9f1e WatchSource:0}: Error finding container 68285cd42beb69e36b2909d32440b9e28e68e8cd105a90198fb225921eec9f1e: Status 404 returned error can't find the container with id 68285cd42beb69e36b2909d32440b9e28e68e8cd105a90198fb225921eec9f1e Sep 09 20:03:30.381731 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:30.381699 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" event={"ID":"98f942f6-35a2-42df-97fe-698ccf44bc2a","Type":"ContainerStarted","Data":"68285cd42beb69e36b2909d32440b9e28e68e8cd105a90198fb225921eec9f1e"} Sep 09 20:03:34.405084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:34.405045 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" event={"ID":"98f942f6-35a2-42df-97fe-698ccf44bc2a","Type":"ContainerStarted","Data":"76b61a4dd0800a58f6844a5b7c2f3cb84bbd5eae15a6e03482a3195107d99ccc"} Sep 09 20:03:34.405521 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:34.405133 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:38.088192 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.088136 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" podStartSLOduration=6.064882266 podStartE2EDuration="10.088115675s" totalImagesPullingTime="4.023233409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:34.429453477 +0000 UTC m=+500.948333192" watchObservedRunningTime="2026-09-09 20:03:38.088115675 +0000 UTC m=+504.606995389" Sep 09 20:03:38.088902 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.088885 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb"] Sep 09 20:03:38.092120 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.092102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:38.096528 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.096512 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-khl4x\"" Sep 09 20:03:38.104392 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.104369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb"] Sep 09 20:03:38.257410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.257350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkfv\" (UniqueName: \"kubernetes.io/projected/bb4ded8d-d639-4cf7-b784-2236cd708580-kube-api-access-gbkfv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-28ctb\" (UID: \"bb4ded8d-d639-4cf7-b784-2236cd708580\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:38.358719 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.358632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkfv\" (UniqueName: \"kubernetes.io/projected/bb4ded8d-d639-4cf7-b784-2236cd708580-kube-api-access-gbkfv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-28ctb\" (UID: \"bb4ded8d-d639-4cf7-b784-2236cd708580\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:38.374708 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.374680 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkfv\" (UniqueName: \"kubernetes.io/projected/bb4ded8d-d639-4cf7-b784-2236cd708580-kube-api-access-gbkfv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-28ctb\" (UID: \"bb4ded8d-d639-4cf7-b784-2236cd708580\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:38.402490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.402456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:38.682703 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:38.682628 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb"] Sep 09 20:03:38.686185 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:03:38.686158 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4ded8d_d639_4cf7_b784_2236cd708580.slice/crio-bcb3ae05aeb546f6819f30cc6cbfe54d9037afff21e217e3176f18ff91633b14 WatchSource:0}: Error finding container bcb3ae05aeb546f6819f30cc6cbfe54d9037afff21e217e3176f18ff91633b14: Status 404 returned error can't find the container with id bcb3ae05aeb546f6819f30cc6cbfe54d9037afff21e217e3176f18ff91633b14 Sep 09 20:03:39.430462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:39.430429 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" event={"ID":"bb4ded8d-d639-4cf7-b784-2236cd708580","Type":"ContainerStarted","Data":"bcb3ae05aeb546f6819f30cc6cbfe54d9037afff21e217e3176f18ff91633b14"} Sep 09 20:03:41.441867 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:41.441828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" event={"ID":"bb4ded8d-d639-4cf7-b784-2236cd708580","Type":"ContainerStarted","Data":"986df6cabc07496ac0519dee0ce2e3d7a772d8f003ca3bea4276f638e53de195"} Sep 09 20:03:41.442380 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:41.441892 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:03:41.461485 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:41.461404 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" podStartSLOduration=1.45531942 podStartE2EDuration="3.461382078s" totalImagesPullingTime="2.006062658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:41.459563099 +0000 UTC m=+507.978442811" watchObservedRunningTime="2026-09-09 20:03:41.461382078 +0000 UTC m=+507.980261792" Sep 09 20:03:45.413066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:45.413034 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-5rnjx" Sep 09 20:03:52.449414 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:03:52.449378 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-28ctb" Sep 09 20:04:24.273014 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.272927 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sksm8"] Sep 09 20:04:24.277322 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.277299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.279412 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.279393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 20:04:24.279508 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.279489 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zkj67\"" Sep 09 20:04:24.286114 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.286087 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sksm8"] Sep 09 20:04:24.319709 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.319677 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sksm8"] Sep 09 20:04:24.369163 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.369126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xn6l\" (UniqueName: \"kubernetes.io/projected/60baa3dd-8dbb-4e9c-812e-955872f82cd3-kube-api-access-4xn6l\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.369359 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.369196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60baa3dd-8dbb-4e9c-812e-955872f82cd3-config-file\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.470498 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.470464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60baa3dd-8dbb-4e9c-812e-955872f82cd3-config-file\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.470719 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.470541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xn6l\" (UniqueName: \"kubernetes.io/projected/60baa3dd-8dbb-4e9c-812e-955872f82cd3-kube-api-access-4xn6l\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.471123 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.471099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60baa3dd-8dbb-4e9c-812e-955872f82cd3-config-file\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.479551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.479493 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xn6l\" (UniqueName: \"kubernetes.io/projected/60baa3dd-8dbb-4e9c-812e-955872f82cd3-kube-api-access-4xn6l\") pod \"limitador-limitador-84994f7ffc-sksm8\" (UID: \"60baa3dd-8dbb-4e9c-812e-955872f82cd3\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.588866 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.588833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:24.870390 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:24.870329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sksm8"] Sep 09 20:04:24.873358 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:04:24.873333 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60baa3dd_8dbb_4e9c_812e_955872f82cd3.slice/crio-04e9570b5e7e8ef15d1c8cff3fb1ef7f9dfd693cf17d204cd199e179404883af WatchSource:0}: Error finding container 04e9570b5e7e8ef15d1c8cff3fb1ef7f9dfd693cf17d204cd199e179404883af: Status 404 returned error can't find the container with id 04e9570b5e7e8ef15d1c8cff3fb1ef7f9dfd693cf17d204cd199e179404883af Sep 09 20:04:25.652876 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:25.652848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" event={"ID":"60baa3dd-8dbb-4e9c-812e-955872f82cd3","Type":"ContainerStarted","Data":"04e9570b5e7e8ef15d1c8cff3fb1ef7f9dfd693cf17d204cd199e179404883af"} Sep 09 20:04:26.659839 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:26.659803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" event={"ID":"60baa3dd-8dbb-4e9c-812e-955872f82cd3","Type":"ContainerStarted","Data":"71f68e0ee5a39b996663908ed323e62b3c281f4ef08550c9616eab610921f687"} Sep 09 20:04:26.660251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:26.659880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:04:26.680781 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:26.680733 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" podStartSLOduration=1.353134334 podStartE2EDuration="2.680715199s" totalImagesPullingTime="1.327580865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:26.680172669 +0000 UTC m=+553.199052381" watchObservedRunningTime="2026-09-09 20:04:26.680715199 +0000 UTC m=+553.199595045" Sep 09 20:04:37.665168 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:04:37.665130 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sksm8" Sep 09 20:06:58.651469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.651439 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8phv8"] Sep 09 20:06:58.655028 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.655008 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8phv8" Sep 09 20:06:58.658388 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.658368 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 09 20:06:58.658507 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.658393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 09 20:06:58.659156 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.659138 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hg5lf\"" Sep 09 20:06:58.659344 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.659331 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 09 20:06:58.659401 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.659362 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 09 20:06:58.682237 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.682193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8phv8"] Sep 09 20:06:58.754168 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.754129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.754401 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.754287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9sl\" (UniqueName: \"kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.855651 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.855609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9sl\" (UniqueName: \"kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.855861 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.855714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.856133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.856110 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.866630 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.866602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9sl\" (UniqueName: \"kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl\") pod \"s3-init-8phv8\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " pod="kserve/s3-init-8phv8" Sep 09 20:06:58.964565 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:58.964482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8phv8" Sep 09 20:06:59.240863 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:59.240788 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8phv8"] Sep 09 20:06:59.244530 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:06:59.244505 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode28634fd_c46f_47b4_ad53_6e8391e17d43.slice/crio-0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5 WatchSource:0}: Error finding container 0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5: Status 404 returned error can't find the container with id 0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5 Sep 09 20:06:59.246568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:59.246549 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:06:59.401740 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:06:59.401705 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerStarted","Data":"0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5"} Sep 09 20:07:03.435338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:07:03.435303 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerStarted","Data":"abbcfc0b1cf779cc63ae17356c2d0a4188c29b99f2a0abc17338b7a60f1db4f6"} Sep 09 20:07:05.445939 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:07:05.445910 2573 generic.go:356] "Generic (PLEG): container finished" podID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerID="abbcfc0b1cf779cc63ae17356c2d0a4188c29b99f2a0abc17338b7a60f1db4f6" exitCode=0 Sep 09 20:07:05.446337 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:07:05.445987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerDied","Data":"abbcfc0b1cf779cc63ae17356c2d0a4188c29b99f2a0abc17338b7a60f1db4f6"} Sep 09 20:07:10.479956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:07:10.479915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerStarted","Data":"437a6ee25f49d37e3c1394d3c25260789b6d03015a4cc300cfe36a0cd50a288d"} Sep 09 20:07:10.500274 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:07:10.500197 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-8phv8" podStartSLOduration=2.500170409 podStartE2EDuration="12.500180076s" totalImagesPullingTime="8.194172595s" totalInitContainerRuntime="1.805837072s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:10.498061281 +0000 UTC m=+717.016940997" watchObservedRunningTime="2026-09-09 20:07:10.500180076 +0000 UTC m=+717.019059788" Sep 09 20:08:15.794717 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:15.794679 2573 generic.go:356] "Generic (PLEG): container finished" podID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerID="437a6ee25f49d37e3c1394d3c25260789b6d03015a4cc300cfe36a0cd50a288d" exitCode=0 Sep 09 20:08:15.795139 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:15.794753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerDied","Data":"437a6ee25f49d37e3c1394d3c25260789b6d03015a4cc300cfe36a0cd50a288d"} Sep 09 20:08:16.928073 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:16.928047 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8phv8" Sep 09 20:08:16.982373 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:16.982341 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache\") pod \"e28634fd-c46f-47b4-ad53-6e8391e17d43\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " Sep 09 20:08:16.982535 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:16.982384 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd9sl\" (UniqueName: \"kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl\") pod \"e28634fd-c46f-47b4-ad53-6e8391e17d43\" (UID: \"e28634fd-c46f-47b4-ad53-6e8391e17d43\") " Sep 09 20:08:16.984724 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:16.984699 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl" (OuterVolumeSpecName: "kube-api-access-sd9sl") pod "e28634fd-c46f-47b4-ad53-6e8391e17d43" (UID: "e28634fd-c46f-47b4-ad53-6e8391e17d43"). InnerVolumeSpecName "kube-api-access-sd9sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:08:17.000910 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.000878 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache" (OuterVolumeSpecName: "model-cache") pod "e28634fd-c46f-47b4-ad53-6e8391e17d43" (UID: "e28634fd-c46f-47b4-ad53-6e8391e17d43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:08:17.083701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.083606 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e28634fd-c46f-47b4-ad53-6e8391e17d43-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:08:17.083701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.083636 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9sl\" (UniqueName: \"kubernetes.io/projected/e28634fd-c46f-47b4-ad53-6e8391e17d43-kube-api-access-sd9sl\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:08:17.806136 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.806106 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8phv8" event={"ID":"e28634fd-c46f-47b4-ad53-6e8391e17d43","Type":"ContainerDied","Data":"0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5"} Sep 09 20:08:17.806136 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.806143 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9f683b9146b13439d9b6c9062914612230640fa71daa303d1aa89ee33439b5" Sep 09 20:08:17.806364 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:17.806117 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8phv8" Sep 09 20:08:59.645259 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645210 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:08:59.645665 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645625 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerName="download-hf-model" Sep 09 20:08:59.645665 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645640 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerName="download-hf-model" Sep 09 20:08:59.645665 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645657 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerName="s3-init" Sep 09 20:08:59.645665 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645663 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerName="s3-init" Sep 09 20:08:59.645806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.645726 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28634fd-c46f-47b4-ad53-6e8391e17d43" containerName="s3-init" Sep 09 20:08:59.649129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.649109 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.653841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.653820 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:08:59.653841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.653826 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\"" Sep 09 20:08:59.654039 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.653872 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-2zp4s\"" Sep 09 20:08:59.654039 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.653979 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\"" Sep 09 20:08:59.654192 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.654173 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\"" Sep 09 20:08:59.657490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.657472 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\"" Sep 09 20:08:59.668862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.668788 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:08:59.755795 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.755795 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79wr\" (UniqueName: \"kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.755976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.756020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.756203 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.756060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857515 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79wr\" (UniqueName: \"kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857939 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.857939 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.857741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.858107 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.858079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.858195 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.858121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.858195 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.858152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.858336 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.858253 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.858392 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.858364 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.860088 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.860067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.860213 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.860138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.866862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.866837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79wr\" (UniqueName: \"kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:08:59.960095 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:08:59.959999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:09:00.243105 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:00.243077 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:09:00.244759 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:09:00.244730 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f31ee9_6aa5_4b18_b1f8_fd3b95052c8f.slice/crio-329e44ee6c82d297e4f1bac3ea691c485c41530d09e0f52d75b7311f7a5a007b WatchSource:0}: Error finding container 329e44ee6c82d297e4f1bac3ea691c485c41530d09e0f52d75b7311f7a5a007b: Status 404 returned error can't find the container with id 329e44ee6c82d297e4f1bac3ea691c485c41530d09e0f52d75b7311f7a5a007b Sep 09 20:09:01.015450 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:01.015392 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerStarted","Data":"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076"} Sep 09 20:09:01.015450 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:01.015463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerStarted","Data":"329e44ee6c82d297e4f1bac3ea691c485c41530d09e0f52d75b7311f7a5a007b"} Sep 09 20:09:03.030205 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:03.030105 2573 generic.go:356] "Generic (PLEG): container finished" podID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerID="b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076" exitCode=0 Sep 09 20:09:03.030205 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:03.030185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerDied","Data":"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076"} Sep 09 20:09:32.217803 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:32.217758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerStarted","Data":"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07"} Sep 09 20:09:32.241183 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:32.241127 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podStartSLOduration=2.6303469489999998 podStartE2EDuration="33.241109412s" totalImagesPullingTime="28.339188445s" totalInitContainerRuntime="2.271574018s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:09:32.239615676 +0000 UTC m=+858.758495389" watchObservedRunningTime="2026-09-09 20:09:32.241109412 +0000 UTC m=+858.759989126" Sep 09 20:09:32.961071 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:32.961036 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:09:39.960866 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:39.960824 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:09:39.962338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:39.962295 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:09:49.960508 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:49.960459 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:09:59.961251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:09:59.961190 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:09.961494 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:09.961389 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:19.960472 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:19.960421 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:29.960462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:29.960419 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:39.961268 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:39.961203 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:49.960635 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:49.960591 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:10:59.960851 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:10:59.960802 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:11:09.960744 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:09.960695 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:11:19.960927 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:19.960877 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:11:29.960958 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:29.960916 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:11:39.970539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:39.970504 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:11:39.979082 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:39.979053 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:11:44.680809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:44.680773 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:11:59.703908 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:59.703844 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" containerID="cri-o://a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07" gracePeriod=45 Sep 09 20:11:59.971649 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:11:59.971542 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Sep 09 20:12:00.458478 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.458453 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:12:00.617614 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617526 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617614 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n79wr\" (UniqueName: \"kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617614 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617592 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617614 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617613 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617645 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617725 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617780 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir\") pod \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\" (UID: \"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f\") " Sep 09 20:12:00.617954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.617908 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:00.618254 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.618152 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.618493 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.618388 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home" (OuterVolumeSpecName: "home") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:00.618493 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.618405 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:12:00.620162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.620134 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:12:00.620347 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.620173 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr" (OuterVolumeSpecName: "kube-api-access-n79wr") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "kube-api-access-n79wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:12:00.620347 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.620198 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm" (OuterVolumeSpecName: "dshm") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:00.638304 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.638274 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:00.646656 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.646625 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" (UID: "b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:00.719646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719615 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n79wr\" (UniqueName: \"kubernetes.io/projected/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kube-api-access-n79wr\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.719646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719642 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.720025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719655 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.720025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719663 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.720025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719672 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.720025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719681 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:00.720025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:00.719688 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:01.003086 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.003001 2573 generic.go:356] "Generic (PLEG): container finished" podID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerID="a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07" exitCode=0 Sep 09 20:12:01.003086 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.003061 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" Sep 09 20:12:01.003292 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.003094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerDied","Data":"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07"} Sep 09 20:12:01.003292 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.003158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7" event={"ID":"b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f","Type":"ContainerDied","Data":"329e44ee6c82d297e4f1bac3ea691c485c41530d09e0f52d75b7311f7a5a007b"} Sep 09 20:12:01.003292 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.003191 2573 scope.go:117] "RemoveContainer" containerID="a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07" Sep 09 20:12:01.013526 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.013506 2573 scope.go:117] "RemoveContainer" containerID="b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076" Sep 09 20:12:01.024881 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.024858 2573 scope.go:117] "RemoveContainer" containerID="a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07" Sep 09 20:12:01.025349 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:12:01.025322 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07\": container with ID starting with a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07 not found: ID does not exist" containerID="a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07" Sep 09 20:12:01.025463 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.025361 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07"} err="failed to get container status \"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07\": rpc error: code = NotFound desc = could not find container \"a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07\": container with ID starting with a8a144919782bfe18becd8b0225a77b42759c271e36a7bf6dd41435f4ba27d07 not found: ID does not exist" Sep 09 20:12:01.025463 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.025388 2573 scope.go:117] "RemoveContainer" containerID="b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076" Sep 09 20:12:01.025685 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:12:01.025665 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076\": container with ID starting with b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076 not found: ID does not exist" containerID="b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076" Sep 09 20:12:01.025737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.025693 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076"} err="failed to get container status \"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076\": rpc error: code = NotFound desc = could not find container \"b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076\": container with ID starting with b499758269eca8cadc853cc841d54a3e040a13f0d2fc62eace33e681f5862076 not found: ID does not exist" Sep 09 20:12:01.026512 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.026420 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:12:01.028772 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:01.028749 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c5dc4dd97-227r7"] Sep 09 20:12:02.014420 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:02.014387 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" path="/var/lib/kubelet/pods/b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f/volumes" Sep 09 20:12:05.922876 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.922834 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:05.923308 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.923290 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="storage-initializer" Sep 09 20:12:05.923308 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.923303 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="storage-initializer" Sep 09 20:12:05.923397 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.923329 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" Sep 09 20:12:05.923397 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.923338 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" Sep 09 20:12:05.923472 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.923399 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f31ee9-6aa5-4b18-b1f8-fd3b95052c8f" containerName="main" Sep 09 20:12:05.926559 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.926539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.929006 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.928985 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\"" Sep 09 20:12:05.929807 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.929785 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-4d9nq\"" Sep 09 20:12:05.929914 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.929828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\"" Sep 09 20:12:05.929914 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.929840 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Sep 09 20:12:05.937484 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.937462 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:05.965346 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l745\" (UniqueName: \"kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.965513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.965513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.965513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.965513 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:05.965647 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:05.965520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066240 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l745\" (UniqueName: \"kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066793 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066793 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.066906 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.066837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.068753 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.068731 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.069096 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.069079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.076275 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.076249 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l745\" (UniqueName: \"kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.238216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.238129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:06.513604 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.513582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:06.516042 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:12:06.516016 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fa0a79_cf95_4798_9db6_b8347632ea1b.slice/crio-97e032542a9036279522628bc7aaeadf47d2c57fd2ecd20ca48e9c3fefd43eaa WatchSource:0}: Error finding container 97e032542a9036279522628bc7aaeadf47d2c57fd2ecd20ca48e9c3fefd43eaa: Status 404 returned error can't find the container with id 97e032542a9036279522628bc7aaeadf47d2c57fd2ecd20ca48e9c3fefd43eaa Sep 09 20:12:06.518075 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:06.518057 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:12:07.043266 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:07.043202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" event={"ID":"d8fa0a79-cf95-4798-9db6-b8347632ea1b","Type":"ContainerStarted","Data":"97e032542a9036279522628bc7aaeadf47d2c57fd2ecd20ca48e9c3fefd43eaa"} Sep 09 20:12:09.054998 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:09.054962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" event={"ID":"d8fa0a79-cf95-4798-9db6-b8347632ea1b","Type":"ContainerStarted","Data":"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991"} Sep 09 20:12:09.074089 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:09.074024 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" podStartSLOduration=2.5841413859999998 podStartE2EDuration="4.074007261s" totalImagesPullingTime="1.489865875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:09.071594638 +0000 UTC m=+1015.590474352" watchObservedRunningTime="2026-09-09 20:12:09.074007261 +0000 UTC m=+1015.592886973" Sep 09 20:12:09.239079 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:09.239043 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:13.226281 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.226244 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:13.235296 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:12:13.235264 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:12:13.235296 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:12:13.235296 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:12:13.235296 ip-10-0-141-196 kubenswrapper[2573]: > containerID="cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" cmd=["/bin/sleep","15"] Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.235313 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" message="" err=< Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:12:13.235326 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerName="main" containerID="cri-o://cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" Sep 09 20:12:13.235695 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.235364 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerName="main" containerID="cri-o://cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" gracePeriod=60 Sep 09 20:12:13.580440 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.580408 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:13.638885 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.638850 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.638885 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.638889 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l745\" (UniqueName: \"kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.639133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.638943 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.639133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.638971 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.639133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.638987 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.639133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.639014 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm\") pod \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\" (UID: \"d8fa0a79-cf95-4798-9db6-b8347632ea1b\") " Sep 09 20:12:13.639368 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.639270 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:13.639368 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.639290 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:13.639368 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.639306 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home" (OuterVolumeSpecName: "home") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:13.641403 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.641372 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm" (OuterVolumeSpecName: "dshm") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:12:13.641403 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.641389 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:12:13.641547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.641405 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745" (OuterVolumeSpecName: "kube-api-access-4l745") pod "d8fa0a79-cf95-4798-9db6-b8347632ea1b" (UID: "d8fa0a79-cf95-4798-9db6-b8347632ea1b"). InnerVolumeSpecName "kube-api-access-4l745". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:12:13.740165 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740116 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:13.740165 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740159 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:13.740165 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740168 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:13.740165 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740175 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fa0a79-cf95-4798-9db6-b8347632ea1b-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:13.740165 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740182 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fa0a79-cf95-4798-9db6-b8347632ea1b-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:13.740501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:13.740189 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l745\" (UniqueName: \"kubernetes.io/projected/d8fa0a79-cf95-4798-9db6-b8347632ea1b-kube-api-access-4l745\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:12:14.081212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.081169 2573 generic.go:356] "Generic (PLEG): container finished" podID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerID="cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" exitCode=0 Sep 09 20:12:14.081391 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.081259 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" Sep 09 20:12:14.081391 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.081266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" event={"ID":"d8fa0a79-cf95-4798-9db6-b8347632ea1b","Type":"ContainerDied","Data":"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991"} Sep 09 20:12:14.081391 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.081366 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f" event={"ID":"d8fa0a79-cf95-4798-9db6-b8347632ea1b","Type":"ContainerDied","Data":"97e032542a9036279522628bc7aaeadf47d2c57fd2ecd20ca48e9c3fefd43eaa"} Sep 09 20:12:14.081538 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.081401 2573 scope.go:117] "RemoveContainer" containerID="cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" Sep 09 20:12:14.090139 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.090116 2573 scope.go:117] "RemoveContainer" containerID="cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" Sep 09 20:12:14.090477 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:12:14.090453 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991\": container with ID starting with cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991 not found: ID does not exist" containerID="cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991" Sep 09 20:12:14.090586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.090483 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991"} err="failed to get container status \"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991\": rpc error: code = NotFound desc = could not find container \"cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991\": container with ID starting with cc9eedf9b8e7affb18ab62dd054d0cc7f1a0de37473b25df865343231a8d5991 not found: ID does not exist" Sep 09 20:12:14.098702 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.098673 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:14.102601 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:14.102576 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-b9l8f"] Sep 09 20:12:16.015208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:16.015110 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" path="/var/lib/kubelet/pods/d8fa0a79-cf95-4798-9db6-b8347632ea1b/volumes" Sep 09 20:12:24.072826 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.072787 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:12:24.073266 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.073210 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerName="main" Sep 09 20:12:24.073266 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.073238 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerName="main" Sep 09 20:12:24.073407 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.073326 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8fa0a79-cf95-4798-9db6-b8347632ea1b" containerName="main" Sep 09 20:12:24.077867 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.077846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.080361 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.080334 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\"" Sep 09 20:12:24.081069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081042 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\"" Sep 09 20:12:24.081069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081059 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" Sep 09 20:12:24.081069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081042 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-t6hcr\"" Sep 09 20:12:24.081383 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081099 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-kwgcw\"" Sep 09 20:12:24.081383 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081141 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\"" Sep 09 20:12:24.081383 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.081043 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:12:24.089342 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.089319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.142821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.142882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.142910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgmn\" (UniqueName: \"kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.142962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.142989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.143107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.143137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.143547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.143158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.243646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.243844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.243844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgmn\" (UniqueName: \"kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.243844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.243844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244099 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244099 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.243977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244099 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244099 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244348 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244409 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244349 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244446 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.244553 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.244535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.246298 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.246272 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.246742 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.246720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.252928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.252902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgmn\" (UniqueName: \"kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.390216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.390114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:24.672562 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:24.672529 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:12:24.675698 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:12:24.675673 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4af454_cf8e_4635_af98_6576f659e046.slice/crio-79c5fb31cc7dc54af3d5d5c4139795542d7bd5331b90f95aa86909bd990e31d9 WatchSource:0}: Error finding container 79c5fb31cc7dc54af3d5d5c4139795542d7bd5331b90f95aa86909bd990e31d9: Status 404 returned error can't find the container with id 79c5fb31cc7dc54af3d5d5c4139795542d7bd5331b90f95aa86909bd990e31d9 Sep 09 20:12:25.137260 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:25.137192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerStarted","Data":"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa"} Sep 09 20:12:25.137742 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:25.137279 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerStarted","Data":"79c5fb31cc7dc54af3d5d5c4139795542d7bd5331b90f95aa86909bd990e31d9"} Sep 09 20:12:27.147850 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:27.147815 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a4af454-cf8e-4635-af98-6576f659e046" containerID="f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa" exitCode=0 Sep 09 20:12:27.148271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:27.147869 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerDied","Data":"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa"} Sep 09 20:12:28.156475 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:28.156439 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerStarted","Data":"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d"} Sep 09 20:12:28.179461 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:28.179395 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.993003708 podStartE2EDuration="4.17937756s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.186373852s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:28.177537159 +0000 UTC m=+1034.696416886" watchObservedRunningTime="2026-09-09 20:12:28.17937756 +0000 UTC m=+1034.698257273" Sep 09 20:12:28.391295 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:28.391254 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:34.390747 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:34.390706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:12:34.392121 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:34.392091 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:12:44.390976 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:44.390929 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:12:54.391512 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:12:54.391474 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:04.390868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:04.390824 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:14.390605 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:14.390560 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:24.390990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:24.390938 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:34.391327 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:34.391286 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:44.390975 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:44.390925 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:13:54.391506 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:13:54.391454 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:14:04.390962 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:04.390915 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:14:14.401205 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:14.401163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:14:14.408873 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:14.408842 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:14:20.695156 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:20.695119 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:14:35.726978 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:35.726856 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" containerID="cri-o://19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d" gracePeriod=45 Sep 09 20:14:36.402280 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.402205 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Sep 09 20:14:36.494856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.494825 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:14:36.554446 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554406 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554446 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554455 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554524 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554542 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554558 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554575 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.554706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554617 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgmn\" (UniqueName: \"kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn\") pod \"5a4af454-cf8e-4635-af98-6576f659e046\" (UID: \"5a4af454-cf8e-4635-af98-6576f659e046\") " Sep 09 20:14:36.555027 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.554922 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:14:36.555151 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.555112 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:14:36.555151 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.555145 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home" (OuterVolumeSpecName: "home") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:14:36.557148 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.557123 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm" (OuterVolumeSpecName: "dshm") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:14:36.557410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.557272 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:14:36.557480 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.557408 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn" (OuterVolumeSpecName: "kube-api-access-9vgmn") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "kube-api-access-9vgmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:14:36.567864 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.567820 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:14:36.578097 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.577986 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a4af454-cf8e-4635-af98-6576f659e046" (UID: "5a4af454-cf8e-4635-af98-6576f659e046"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:14:36.656234 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656173 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656234 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656215 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4af454-cf8e-4635-af98-6576f659e046-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656255 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656266 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656274 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656282 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af454-cf8e-4635-af98-6576f659e046-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656290 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4af454-cf8e-4635-af98-6576f659e046-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.656497 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.656297 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgmn\" (UniqueName: \"kubernetes.io/projected/5a4af454-cf8e-4635-af98-6576f659e046-kube-api-access-9vgmn\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:14:36.874986 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.874890 2573 generic.go:356] "Generic (PLEG): container finished" podID="5a4af454-cf8e-4635-af98-6576f659e046" containerID="19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d" exitCode=0 Sep 09 20:14:36.874986 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.874934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerDied","Data":"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d"} Sep 09 20:14:36.874986 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.874948 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Sep 09 20:14:36.874986 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.874981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"5a4af454-cf8e-4635-af98-6576f659e046","Type":"ContainerDied","Data":"79c5fb31cc7dc54af3d5d5c4139795542d7bd5331b90f95aa86909bd990e31d9"} Sep 09 20:14:36.875559 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.875003 2573 scope.go:117] "RemoveContainer" containerID="19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d" Sep 09 20:14:36.886625 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.886599 2573 scope.go:117] "RemoveContainer" containerID="f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa" Sep 09 20:14:36.895366 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.895334 2573 scope.go:117] "RemoveContainer" containerID="19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d" Sep 09 20:14:36.895833 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:14:36.895808 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d\": container with ID starting with 19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d not found: ID does not exist" containerID="19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d" Sep 09 20:14:36.895985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.895845 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d"} err="failed to get container status \"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d\": rpc error: code = NotFound desc = could not find container \"19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d\": container with ID starting with 19274008836f5f97717e63cd2cc691112bd0abf4165833e0b94cbec835499a1d not found: ID does not exist" Sep 09 20:14:36.895985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.895875 2573 scope.go:117] "RemoveContainer" containerID="f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa" Sep 09 20:14:36.896303 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:14:36.896273 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa\": container with ID starting with f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa not found: ID does not exist" containerID="f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa" Sep 09 20:14:36.896434 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.896312 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa"} err="failed to get container status \"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa\": rpc error: code = NotFound desc = could not find container \"f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa\": container with ID starting with f8838fbe1f963910cf29af981dfbaa92ddcf6c505bfb1541bc7116d617a41aaa not found: ID does not exist" Sep 09 20:14:36.898356 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.898331 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:14:36.901825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:36.901803 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Sep 09 20:14:38.014902 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:14:38.014869 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4af454-cf8e-4635-af98-6576f659e046" path="/var/lib/kubelet/pods/5a4af454-cf8e-4635-af98-6576f659e046/volumes" Sep 09 20:15:15.732394 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732357 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:15.732806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732762 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" Sep 09 20:15:15.732806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732772 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" Sep 09 20:15:15.732806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732784 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="storage-initializer" Sep 09 20:15:15.732806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732789 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="storage-initializer" Sep 09 20:15:15.732936 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.732848 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4af454-cf8e-4635-af98-6576f659e046" containerName="main" Sep 09 20:15:15.737104 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.737084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.740467 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.740441 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\"" Sep 09 20:15:15.740592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.740448 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\"" Sep 09 20:15:15.740592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.740450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" Sep 09 20:15:15.740592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.740469 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-krtpk\"" Sep 09 20:15:15.744470 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.744428 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:15.795996 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.795954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.795996 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.796006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74rv\" (UniqueName: \"kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.796243 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.796044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.796243 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.796120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.796243 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.796163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.796243 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.796185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.897720 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.897686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.897886 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.897731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.897886 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.897757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.897990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.897888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.898032 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.897991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.898086 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.898035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74rv\" (UniqueName: \"kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.898145 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.898131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.898206 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.898189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.898408 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.898382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.900288 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.900265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.900420 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.900402 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:15.906694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:15.906667 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74rv\" (UniqueName: \"kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv\") pod \"scheduler-inline-config-test-kserve-8655bc757-wvv4t\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:16.048239 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:16.048195 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:16.322867 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:16.322838 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:16.325545 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:15:16.325519 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ad47e9_acd1_4b53_8e00_8ab197cbe495.slice/crio-3c9f356026bd01f4f4838179065097b3938e288e20d8de821ab805f6dd2a3efd WatchSource:0}: Error finding container 3c9f356026bd01f4f4838179065097b3938e288e20d8de821ab805f6dd2a3efd: Status 404 returned error can't find the container with id 3c9f356026bd01f4f4838179065097b3938e288e20d8de821ab805f6dd2a3efd Sep 09 20:15:17.080265 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:17.080213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" event={"ID":"86ad47e9-acd1-4b53-8e00-8ab197cbe495","Type":"ContainerStarted","Data":"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a"} Sep 09 20:15:17.080649 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:17.080290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" event={"ID":"86ad47e9-acd1-4b53-8e00-8ab197cbe495","Type":"ContainerStarted","Data":"3c9f356026bd01f4f4838179065097b3938e288e20d8de821ab805f6dd2a3efd"} Sep 09 20:15:17.110025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:17.109965 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" podStartSLOduration=2.109941303 podStartE2EDuration="2.109941303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:17.108875652 +0000 UTC m=+1203.627755376" watchObservedRunningTime="2026-09-09 20:15:17.109941303 +0000 UTC m=+1203.628821017" Sep 09 20:15:18.048618 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:18.048585 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:26.049426 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:26.049389 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:26.061844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:26.061818 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:26.135363 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:26.135335 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:50.611143 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:50.611108 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:50.621263 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:15:50.621201 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:15:50.621263 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:15:50.621263 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:15:50.621263 ip-10-0-141-196 kubenswrapper[2573]: > containerID="04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" cmd=["/bin/sleep","15"] Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:50.621277 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" message="" err=< Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:15:50.621291 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerName="main" containerID="cri-o://04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" Sep 09 20:15:50.621532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:50.621340 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerName="main" containerID="cri-o://04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" gracePeriod=60 Sep 09 20:15:50.987286 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:50.987260 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:51.129047 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129005 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129253 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129253 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129119 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129253 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129155 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129253 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129180 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m74rv\" (UniqueName: \"kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129451 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129281 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache\") pod \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\" (UID: \"86ad47e9-acd1-4b53-8e00-8ab197cbe495\") " Sep 09 20:15:51.129451 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:15:51.129567 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129445 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home" (OuterVolumeSpecName: "home") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:15:51.129639 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129608 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache" (OuterVolumeSpecName: "model-cache") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:15:51.129990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129792 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.129990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129815 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.129990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.129830 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.131646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.131617 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:15:51.132060 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.132037 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm" (OuterVolumeSpecName: "dshm") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:15:51.132060 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.132042 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv" (OuterVolumeSpecName: "kube-api-access-m74rv") pod "86ad47e9-acd1-4b53-8e00-8ab197cbe495" (UID: "86ad47e9-acd1-4b53-8e00-8ab197cbe495"). InnerVolumeSpecName "kube-api-access-m74rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:15:51.231270 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.231155 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ad47e9-acd1-4b53-8e00-8ab197cbe495-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.231270 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.231186 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ad47e9-acd1-4b53-8e00-8ab197cbe495-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.231270 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.231198 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74rv\" (UniqueName: \"kubernetes.io/projected/86ad47e9-acd1-4b53-8e00-8ab197cbe495-kube-api-access-m74rv\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:15:51.254136 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.254104 2573 generic.go:356] "Generic (PLEG): container finished" podID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerID="04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" exitCode=0 Sep 09 20:15:51.254351 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.254149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" Sep 09 20:15:51.254351 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.254195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" event={"ID":"86ad47e9-acd1-4b53-8e00-8ab197cbe495","Type":"ContainerDied","Data":"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a"} Sep 09 20:15:51.254351 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.254277 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t" event={"ID":"86ad47e9-acd1-4b53-8e00-8ab197cbe495","Type":"ContainerDied","Data":"3c9f356026bd01f4f4838179065097b3938e288e20d8de821ab805f6dd2a3efd"} Sep 09 20:15:51.254351 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.254313 2573 scope.go:117] "RemoveContainer" containerID="04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" Sep 09 20:15:51.263988 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.263962 2573 scope.go:117] "RemoveContainer" containerID="04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" Sep 09 20:15:51.264318 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:15:51.264289 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a\": container with ID starting with 04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a not found: ID does not exist" containerID="04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a" Sep 09 20:15:51.264418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.264321 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a"} err="failed to get container status \"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a\": rpc error: code = NotFound desc = could not find container \"04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a\": container with ID starting with 04ffbb5ba48e91a2482d3b0bd8307d77519355dfd5e796cf6c782dbba59a755a not found: ID does not exist" Sep 09 20:15:51.279701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.279661 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:51.281378 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:51.281355 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wvv4t"] Sep 09 20:15:52.015034 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:15:52.014997 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" path="/var/lib/kubelet/pods/86ad47e9-acd1-4b53-8e00-8ab197cbe495/volumes" Sep 09 20:16:56.157301 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.157266 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:16:56.157790 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.157639 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerName="main" Sep 09 20:16:56.157790 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.157650 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerName="main" Sep 09 20:16:56.157790 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.157715 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ad47e9-acd1-4b53-8e00-8ab197cbe495" containerName="main" Sep 09 20:16:56.160779 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.160762 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.163554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.163535 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\"" Sep 09 20:16:56.164783 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.164765 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-jq6fb\"" Sep 09 20:16:56.164876 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.164852 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\"" Sep 09 20:16:56.164919 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.164887 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Sep 09 20:16:56.174394 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.174367 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:16:56.207506 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.207725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.207725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.207725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.207725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.207913 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.207747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpltt\" (UniqueName: \"kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308435 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308872 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpltt\" (UniqueName: \"kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308872 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.308872 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.309006 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.308887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.310924 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.310902 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.311154 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.311136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.317806 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.317783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpltt\" (UniqueName: \"kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.472095 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.472003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:16:56.753670 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:56.753642 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:16:56.754867 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:16:56.754840 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ee5d83_f691_4d2c_b271_e831a2f5d1d0.slice/crio-1aad7da0814ce138b9f5623b4748d9e3c1024948bc02c134295239f1f4be7bb6 WatchSource:0}: Error finding container 1aad7da0814ce138b9f5623b4748d9e3c1024948bc02c134295239f1f4be7bb6: Status 404 returned error can't find the container with id 1aad7da0814ce138b9f5623b4748d9e3c1024948bc02c134295239f1f4be7bb6 Sep 09 20:16:57.582078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:57.582040 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" event={"ID":"62ee5d83-f691-4d2c-b271-e831a2f5d1d0","Type":"ContainerStarted","Data":"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93"} Sep 09 20:16:57.582551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:57.582093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" event={"ID":"62ee5d83-f691-4d2c-b271-e831a2f5d1d0","Type":"ContainerStarted","Data":"1aad7da0814ce138b9f5623b4748d9e3c1024948bc02c134295239f1f4be7bb6"} Sep 09 20:16:57.600947 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:57.600889 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" podStartSLOduration=1.600871688 podStartE2EDuration="1.600871688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:57.599842296 +0000 UTC m=+1304.118722012" watchObservedRunningTime="2026-09-09 20:16:57.600871688 +0000 UTC m=+1304.119751401" Sep 09 20:16:58.472176 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:16:58.472137 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:06.472625 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:06.472576 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:06.485340 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:06.485308 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:06.636093 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:06.636064 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:30.746297 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:30.746266 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:17:30.755564 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:17:30.755523 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:17:30.755564 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:17:30.755564 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:17:30.755564 ip-10-0-141-196 kubenswrapper[2573]: > containerID="a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" cmd=["/bin/sleep","15"] Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:30.755570 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" message="" err=< Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:17:30.755584 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerName="main" containerID="cri-o://a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" Sep 09 20:17:30.755875 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:30.755629 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerName="main" containerID="cri-o://a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" gracePeriod=60 Sep 09 20:17:31.098635 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.098609 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:31.226016 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.225988 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpltt\" (UniqueName: \"kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226241 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226037 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226241 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226099 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226241 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226146 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226241 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226241 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache\") pod \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\" (UID: \"62ee5d83-f691-4d2c-b271-e831a2f5d1d0\") " Sep 09 20:17:31.226529 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226377 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home" (OuterVolumeSpecName: "home") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:17:31.226529 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226415 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:17:31.226632 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226566 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.226632 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226579 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.226632 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.226586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:17:31.228397 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.228365 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm" (OuterVolumeSpecName: "dshm") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:17:31.228525 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.228414 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt" (OuterVolumeSpecName: "kube-api-access-bpltt") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "kube-api-access-bpltt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:17:31.228525 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.228450 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62ee5d83-f691-4d2c-b271-e831a2f5d1d0" (UID: "62ee5d83-f691-4d2c-b271-e831a2f5d1d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:17:31.327350 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.327312 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpltt\" (UniqueName: \"kubernetes.io/projected/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-kube-api-access-bpltt\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.327350 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.327343 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.327350 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.327353 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.327350 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.327363 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee5d83-f691-4d2c-b271-e831a2f5d1d0-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:17:31.753944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.753857 2573 generic.go:356] "Generic (PLEG): container finished" podID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerID="a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" exitCode=0 Sep 09 20:17:31.753944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.753898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" event={"ID":"62ee5d83-f691-4d2c-b271-e831a2f5d1d0","Type":"ContainerDied","Data":"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93"} Sep 09 20:17:31.753944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.753905 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" Sep 09 20:17:31.754582 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.753946 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d" event={"ID":"62ee5d83-f691-4d2c-b271-e831a2f5d1d0","Type":"ContainerDied","Data":"1aad7da0814ce138b9f5623b4748d9e3c1024948bc02c134295239f1f4be7bb6"} Sep 09 20:17:31.754582 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.753971 2573 scope.go:117] "RemoveContainer" containerID="a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" Sep 09 20:17:31.764278 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.764112 2573 scope.go:117] "RemoveContainer" containerID="a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" Sep 09 20:17:31.764485 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:17:31.764467 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93\": container with ID starting with a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93 not found: ID does not exist" containerID="a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93" Sep 09 20:17:31.764539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.764494 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93"} err="failed to get container status \"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93\": rpc error: code = NotFound desc = could not find container \"a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93\": container with ID starting with a2c8f3e441b9da9dacd11cf44296c68c364573f9eab965bdeb95c8062f684d93 not found: ID does not exist" Sep 09 20:17:31.787573 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.787543 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:17:31.797091 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:31.797057 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fzr72d"] Sep 09 20:17:32.015104 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:17:32.015021 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" path="/var/lib/kubelet/pods/62ee5d83-f691-4d2c-b271-e831a2f5d1d0/volumes" Sep 09 20:18:33.041149 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.041107 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:18:33.041783 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.041765 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerName="main" Sep 09 20:18:33.041908 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.041784 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerName="main" Sep 09 20:18:33.041973 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.041946 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ee5d83-f691-4d2c-b271-e831a2f5d1d0" containerName="main" Sep 09 20:18:33.046323 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.046302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.048756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.048735 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\"" Sep 09 20:18:33.048864 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.048735 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\"" Sep 09 20:18:33.048930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.048878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-gbrpr\"" Sep 09 20:18:33.050630 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.050611 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\"" Sep 09 20:18:33.056050 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.056025 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:18:33.074581 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwgg\" (UniqueName: \"kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.074756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.074756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074638 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.074756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.074756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.074895 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.074755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175396 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwgg\" (UniqueName: \"kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.175961 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175940 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.176036 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.175977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.176152 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.176133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.177939 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.177920 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.178307 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.178291 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.184651 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.184631 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwgg\" (UniqueName: \"kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-n5497\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.357972 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.357868 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:33.630761 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:33.630598 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:18:33.633597 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:18:33.633570 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fadf6ba_d2f1_4e57_a156_c5e1ff03ae77.slice/crio-bb261cb8f72d4a2659a704015237c6b645e595e47a78bfc6a6fdff8f22fb58ea WatchSource:0}: Error finding container bb261cb8f72d4a2659a704015237c6b645e595e47a78bfc6a6fdff8f22fb58ea: Status 404 returned error can't find the container with id bb261cb8f72d4a2659a704015237c6b645e595e47a78bfc6a6fdff8f22fb58ea Sep 09 20:18:34.067910 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:34.067877 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" event={"ID":"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77","Type":"ContainerStarted","Data":"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e"} Sep 09 20:18:34.068304 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:34.067932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" event={"ID":"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77","Type":"ContainerStarted","Data":"bb261cb8f72d4a2659a704015237c6b645e595e47a78bfc6a6fdff8f22fb58ea"} Sep 09 20:18:34.088930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:34.088854 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" podStartSLOduration=1.088832221 podStartE2EDuration="1.088832221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:34.088170971 +0000 UTC m=+1400.607050685" watchObservedRunningTime="2026-09-09 20:18:34.088832221 +0000 UTC m=+1400.607711935" Sep 09 20:18:34.358037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:34.357942 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:43.358649 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:43.358613 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:43.371421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:43.371393 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:18:44.127495 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:18:44.127461 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:19:07.139239 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.139144 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:19:07.149825 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:19:07.149784 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:19:07.149825 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:19:07.149825 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:19:07.149825 ip-10-0-141-196 kubenswrapper[2573]: > containerID="a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" cmd=["/bin/sleep","15"] Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.149840 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" message="" err=< Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:19:07.149852 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerName="main" containerID="cri-o://a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" Sep 09 20:19:07.150149 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.149890 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerName="main" containerID="cri-o://a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" gracePeriod=60 Sep 09 20:19:07.498316 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.498292 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:19:07.590021 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.589981 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590215 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mwgg\" (UniqueName: \"kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590215 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590070 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590215 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590091 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590215 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590135 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590215 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590182 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache\") pod \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\" (UID: \"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77\") " Sep 09 20:19:07.590529 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590456 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home" (OuterVolumeSpecName: "home") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:19:07.590529 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590468 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:19:07.590606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.590530 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:19:07.592537 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.592514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm" (OuterVolumeSpecName: "dshm") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:19:07.592663 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.592644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg" (OuterVolumeSpecName: "kube-api-access-9mwgg") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "kube-api-access-9mwgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:19:07.592663 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.592639 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" (UID: "6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691875 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691905 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwgg\" (UniqueName: \"kubernetes.io/projected/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-kube-api-access-9mwgg\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691917 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691928 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691935 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:07.691974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:07.691943 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:19:08.240503 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.240469 2573 generic.go:356] "Generic (PLEG): container finished" podID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerID="a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" exitCode=0 Sep 09 20:19:08.240937 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.240517 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" Sep 09 20:19:08.240937 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.240543 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" event={"ID":"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77","Type":"ContainerDied","Data":"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e"} Sep 09 20:19:08.240937 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.240585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497" event={"ID":"6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77","Type":"ContainerDied","Data":"bb261cb8f72d4a2659a704015237c6b645e595e47a78bfc6a6fdff8f22fb58ea"} Sep 09 20:19:08.240937 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.240607 2573 scope.go:117] "RemoveContainer" containerID="a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" Sep 09 20:19:08.250166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.250148 2573 scope.go:117] "RemoveContainer" containerID="a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" Sep 09 20:19:08.250515 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:19:08.250496 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e\": container with ID starting with a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e not found: ID does not exist" containerID="a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e" Sep 09 20:19:08.250577 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.250525 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e"} err="failed to get container status \"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e\": rpc error: code = NotFound desc = could not find container \"a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e\": container with ID starting with a3ac8767fc43f12a45ec817d7f8fc733459acce13dce7f48e95e15f217479c8e not found: ID does not exist" Sep 09 20:19:08.263817 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.263778 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:19:08.270353 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:08.270326 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-n5497"] Sep 09 20:19:10.015343 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:10.015308 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" path="/var/lib/kubelet/pods/6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77/volumes" Sep 09 20:19:42.459470 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.459433 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:19:42.460065 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.460027 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerName="main" Sep 09 20:19:42.460065 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.460046 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerName="main" Sep 09 20:19:42.460191 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.460133 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fadf6ba-d2f1-4e57-a156-c5e1ff03ae77" containerName="main" Sep 09 20:19:42.463216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.463196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.465925 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.465889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:19:42.466049 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.465953 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\"" Sep 09 20:19:42.466049 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.465967 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-xx7c2\"" Sep 09 20:19:42.466049 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.465984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\"" Sep 09 20:19:42.466049 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.466016 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\"" Sep 09 20:19:42.466338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.466320 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\"" Sep 09 20:19:42.476741 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.476719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:19:42.591809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.591809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zww6\" (UniqueName: \"kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.591982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.592023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.592085 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.592051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.692700 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.692893 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.692893 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.692893 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.692893 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692952 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693118 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.692994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zww6\" (UniqueName: \"kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693328 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.693132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693466 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.693434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693591 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.693514 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693649 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.693597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.693649 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.693601 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.695522 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.695477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.695926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.695905 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.712255 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.712165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zww6\" (UniqueName: \"kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6\") pod \"auth-disabled-test-kserve-54cbbbf45d-scmhb\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:42.774143 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:42.774104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:43.056510 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:43.056479 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:19:43.058422 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:19:43.058389 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299f5e47_7619_4422_9de9_ce2c1ce2b8e2.slice/crio-1974c7a092c619cc0f3b99f32ddb77a9528cbb5417aca969157b6dade707a435 WatchSource:0}: Error finding container 1974c7a092c619cc0f3b99f32ddb77a9528cbb5417aca969157b6dade707a435: Status 404 returned error can't find the container with id 1974c7a092c619cc0f3b99f32ddb77a9528cbb5417aca969157b6dade707a435 Sep 09 20:19:43.411720 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:43.411632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerStarted","Data":"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725"} Sep 09 20:19:43.411720 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:43.411682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerStarted","Data":"1974c7a092c619cc0f3b99f32ddb77a9528cbb5417aca969157b6dade707a435"} Sep 09 20:19:45.424814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:45.424781 2573 generic.go:356] "Generic (PLEG): container finished" podID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerID="2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725" exitCode=0 Sep 09 20:19:45.425189 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:45.424855 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerDied","Data":"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725"} Sep 09 20:19:46.432374 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:46.432341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerStarted","Data":"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a"} Sep 09 20:19:46.456428 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:46.456378 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podStartSLOduration=2.310971126 podStartE2EDuration="4.456358523s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.145387397s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:46.455039844 +0000 UTC m=+1472.973919558" watchObservedRunningTime="2026-09-09 20:19:46.456358523 +0000 UTC m=+1472.975238237" Sep 09 20:19:46.775313 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:46.775208 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:52.774334 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:52.774301 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:19:52.775877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:19:52.775850 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:02.774848 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:02.774807 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:12.774729 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:12.774681 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:14.089051 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.089021 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:20:14.092837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.092814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.096568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.096544 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-m5n7v\"" Sep 09 20:20:14.096705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.096615 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-7kvrs\"" Sep 09 20:20:14.096705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.096653 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\"" Sep 09 20:20:14.096705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.096673 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Sep 09 20:20:14.096705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.096653 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\"" Sep 09 20:20:14.101982 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.101961 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:20:14.194070 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.194037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.194266 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.194102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbml\" (UniqueName: \"kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.294840 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.294798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.295036 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.294892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbml\" (UniqueName: \"kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.297473 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.297450 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.303914 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.303885 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbml\" (UniqueName: \"kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.404016 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.403924 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:14.679309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.679284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:20:14.682541 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:20:14.682514 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20871988_3d4d_4ca2_800b_69cadc7b6e9b.slice/crio-5005592c08dd917fec76dc43a0d274bee86e4be190551f7a943672f4c9ba3fc0 WatchSource:0}: Error finding container 5005592c08dd917fec76dc43a0d274bee86e4be190551f7a943672f4c9ba3fc0: Status 404 returned error can't find the container with id 5005592c08dd917fec76dc43a0d274bee86e4be190551f7a943672f4c9ba3fc0 Sep 09 20:20:14.685244 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:14.685207 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:20:15.587057 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:15.586931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" event={"ID":"20871988-3d4d-4ca2-800b-69cadc7b6e9b","Type":"ContainerStarted","Data":"5005592c08dd917fec76dc43a0d274bee86e4be190551f7a943672f4c9ba3fc0"} Sep 09 20:20:17.607198 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:17.607160 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" event={"ID":"20871988-3d4d-4ca2-800b-69cadc7b6e9b","Type":"ContainerStarted","Data":"bdd8631767dc56b11177a92fb5583a6ac132ca69d808627ed53e7b58b122340c"} Sep 09 20:20:17.607626 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:17.607293 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:20:17.629145 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:17.629069 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podStartSLOduration=0.892930749 podStartE2EDuration="3.629043122s" totalImagesPullingTime="2.736112373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:17.626865215 +0000 UTC m=+1504.145744953" watchObservedRunningTime="2026-09-09 20:20:17.629043122 +0000 UTC m=+1504.147922845" Sep 09 20:20:22.775335 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:22.775298 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:32.775552 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:32.775511 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:42.774963 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:42.774921 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:20:48.615609 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:48.615566 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:20:50.837978 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:50.837932 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:20:50.840767 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:50.840728 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:20:52.775426 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:20:52.775372 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:00.839710 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:00.839662 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:21:02.775577 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:02.775529 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:05.860316 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:05.860213 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" containerID="cri-o://bdd8631767dc56b11177a92fb5583a6ac132ca69d808627ed53e7b58b122340c" gracePeriod=45 Sep 09 20:21:10.840003 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:10.839947 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:21:12.775194 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:12.775147 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:20.839874 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:20.839834 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:21:22.775240 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:22.775190 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:30.840317 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:30.840274 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:21:32.775047 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:32.775010 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:36.041837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.041798 2573 generic.go:356] "Generic (PLEG): container finished" podID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerID="bdd8631767dc56b11177a92fb5583a6ac132ca69d808627ed53e7b58b122340c" exitCode=0 Sep 09 20:21:36.042182 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.041885 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" event={"ID":"20871988-3d4d-4ca2-800b-69cadc7b6e9b","Type":"ContainerDied","Data":"bdd8631767dc56b11177a92fb5583a6ac132ca69d808627ed53e7b58b122340c"} Sep 09 20:21:36.136436 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.136367 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:21:36.177555 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.177514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqbml\" (UniqueName: \"kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml\") pod \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " Sep 09 20:21:36.177747 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.177615 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs\") pod \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\" (UID: \"20871988-3d4d-4ca2-800b-69cadc7b6e9b\") " Sep 09 20:21:36.179918 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.179891 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20871988-3d4d-4ca2-800b-69cadc7b6e9b" (UID: "20871988-3d4d-4ca2-800b-69cadc7b6e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:21:36.180042 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.179981 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml" (OuterVolumeSpecName: "kube-api-access-vqbml") pod "20871988-3d4d-4ca2-800b-69cadc7b6e9b" (UID: "20871988-3d4d-4ca2-800b-69cadc7b6e9b"). InnerVolumeSpecName "kube-api-access-vqbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:21:36.278837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.278810 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbml\" (UniqueName: \"kubernetes.io/projected/20871988-3d4d-4ca2-800b-69cadc7b6e9b-kube-api-access-vqbml\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:21:36.278837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:36.278838 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20871988-3d4d-4ca2-800b-69cadc7b6e9b-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:21:37.048811 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:37.048781 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" Sep 09 20:21:37.048811 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:37.048798 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx" event={"ID":"20871988-3d4d-4ca2-800b-69cadc7b6e9b","Type":"ContainerDied","Data":"5005592c08dd917fec76dc43a0d274bee86e4be190551f7a943672f4c9ba3fc0"} Sep 09 20:21:37.049349 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:37.048854 2573 scope.go:117] "RemoveContainer" containerID="bdd8631767dc56b11177a92fb5583a6ac132ca69d808627ed53e7b58b122340c" Sep 09 20:21:37.078745 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:37.078700 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:21:37.080450 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:37.080426 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5t9rkx"] Sep 09 20:21:38.017457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:38.017424 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" path="/var/lib/kubelet/pods/20871988-3d4d-4ca2-800b-69cadc7b6e9b/volumes" Sep 09 20:21:42.774580 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:42.774538 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Sep 09 20:21:52.785087 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:52.785043 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:21:52.793114 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:21:52.793085 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:22:05.746007 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:05.745925 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:22:20.770283 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:20.770192 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" containerID="cri-o://a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a" gracePeriod=45 Sep 09 20:22:21.525545 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.525519 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:22:21.604273 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604176 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604273 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604216 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604273 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604272 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zww6\" (UniqueName: \"kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604544 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604305 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604544 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604346 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604544 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604544 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604398 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604544 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm\") pod \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\" (UID: \"299f5e47-7619-4422-9de9-ce2c1ce2b8e2\") " Sep 09 20:22:21.604813 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604689 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:22:21.604953 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.604919 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:22:21.605154 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.605130 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home" (OuterVolumeSpecName: "home") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:22:21.607112 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.607082 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm" (OuterVolumeSpecName: "dshm") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:22:21.607246 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.607137 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6" (OuterVolumeSpecName: "kube-api-access-8zww6") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "kube-api-access-8zww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:22:21.607399 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.607371 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:22:21.617709 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.617683 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:22:21.627547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.627516 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "299f5e47-7619-4422-9de9-ce2c1ce2b8e2" (UID: "299f5e47-7619-4422-9de9-ce2c1ce2b8e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:22:21.705150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705111 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705141 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705153 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zww6\" (UniqueName: \"kubernetes.io/projected/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-kube-api-access-8zww6\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705164 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705173 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705180 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705189 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:21.705462 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:21.705198 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f5e47-7619-4422-9de9-ce2c1ce2b8e2-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:22:22.285466 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.285433 2573 generic.go:356] "Generic (PLEG): container finished" podID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerID="a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a" exitCode=0 Sep 09 20:22:22.285896 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.285483 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" Sep 09 20:22:22.285896 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.285528 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerDied","Data":"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a"} Sep 09 20:22:22.285896 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.285585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb" event={"ID":"299f5e47-7619-4422-9de9-ce2c1ce2b8e2","Type":"ContainerDied","Data":"1974c7a092c619cc0f3b99f32ddb77a9528cbb5417aca969157b6dade707a435"} Sep 09 20:22:22.285896 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.285612 2573 scope.go:117] "RemoveContainer" containerID="a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a" Sep 09 20:22:22.294550 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.294527 2573 scope.go:117] "RemoveContainer" containerID="2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725" Sep 09 20:22:22.303129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.303107 2573 scope.go:117] "RemoveContainer" containerID="a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a" Sep 09 20:22:22.303475 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:22:22.303452 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a\": container with ID starting with a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a not found: ID does not exist" containerID="a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a" Sep 09 20:22:22.303586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.303483 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a"} err="failed to get container status \"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a\": rpc error: code = NotFound desc = could not find container \"a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a\": container with ID starting with a5e4d2df3627e5f005a971f8781abcb20472d0d0f0042164c4236aecfec1b83a not found: ID does not exist" Sep 09 20:22:22.303586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.303503 2573 scope.go:117] "RemoveContainer" containerID="2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725" Sep 09 20:22:22.303791 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:22:22.303774 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725\": container with ID starting with 2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725 not found: ID does not exist" containerID="2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725" Sep 09 20:22:22.303835 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.303807 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725"} err="failed to get container status \"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725\": rpc error: code = NotFound desc = could not find container \"2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725\": container with ID starting with 2b77353b8ad6f39aded8b8d24293ce58e3aeacd452f5ba981c2cf97b731c2725 not found: ID does not exist" Sep 09 20:22:22.307640 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.307614 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:22:22.311481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:22.311443 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54cbbbf45d-scmhb"] Sep 09 20:22:24.014718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:22:24.014689 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" path="/var/lib/kubelet/pods/299f5e47-7619-4422-9de9-ce2c1ce2b8e2/volumes" Sep 09 20:23:33.896582 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896550 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896916 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="storage-initializer" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896927 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="storage-initializer" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896938 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896945 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896958 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.896963 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.897040 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="299f5e47-7619-4422-9de9-ce2c1ce2b8e2" containerName="main" Sep 09 20:23:33.897062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.897052 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20871988-3d4d-4ca2-800b-69cadc7b6e9b" containerName="main" Sep 09 20:23:33.901431 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.901413 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:33.904745 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.904719 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\"" Sep 09 20:23:33.904856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.904742 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\"" Sep 09 20:23:33.904856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.904790 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q2x8k\"" Sep 09 20:23:33.904856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.904814 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\"" Sep 09 20:23:33.909797 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:33.909771 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:23:34.077856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.077813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.078051 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.077918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.078051 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.077973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.078051 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.078004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.078216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.078054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.078216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.078081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hngt\" (UniqueName: \"kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179593 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179593 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179829 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179829 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179829 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.179829 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hngt\" (UniqueName: \"kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.180038 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.179928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.180087 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.180041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.180087 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.180061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.182120 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.182094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.182332 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.182318 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.188785 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.188756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hngt\" (UniqueName: \"kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.213282 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.213213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:34.490180 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.490075 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:23:34.493836 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:23:34.493809 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c3ea8c_d99e_4aaa_a60e_4f77a3fdf429.slice/crio-43a882d616092ae7ee1e85f7a09d79a8cdb74024c49e18774349d5299c2b76e6 WatchSource:0}: Error finding container 43a882d616092ae7ee1e85f7a09d79a8cdb74024c49e18774349d5299c2b76e6: Status 404 returned error can't find the container with id 43a882d616092ae7ee1e85f7a09d79a8cdb74024c49e18774349d5299c2b76e6 Sep 09 20:23:34.649782 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.649741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" event={"ID":"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429","Type":"ContainerStarted","Data":"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b"} Sep 09 20:23:34.649970 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.649826 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" event={"ID":"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429","Type":"ContainerStarted","Data":"43a882d616092ae7ee1e85f7a09d79a8cdb74024c49e18774349d5299c2b76e6"} Sep 09 20:23:34.669386 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:34.669332 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" podStartSLOduration=1.66931516 podStartE2EDuration="1.66931516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:34.669096116 +0000 UTC m=+1701.187975824" watchObservedRunningTime="2026-09-09 20:23:34.66931516 +0000 UTC m=+1701.188194872" Sep 09 20:23:35.214312 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:35.214267 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:44.213482 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:44.213438 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:44.225979 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:44.225936 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:23:44.712270 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:23:44.712218 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:25:20.408016 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.407964 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:25:20.418059 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:25:20.418018 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:25:20.418059 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:25:20.418059 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:25:20.418059 ip-10-0-141-196 kubenswrapper[2573]: > containerID="6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" cmd=["/bin/sleep","15"] Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.418079 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" message="" err=< Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:25:20.418093 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" containerID="cri-o://6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" Sep 09 20:25:20.418371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.418136 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" containerID="cri-o://6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" gracePeriod=60 Sep 09 20:25:20.770611 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.770584 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:25:20.873828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.873787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.873828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.873834 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hngt\" (UniqueName: \"kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.874084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.873862 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.874084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.873892 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.874084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874013 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.874300 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874129 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir\") pod \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\" (UID: \"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429\") " Sep 09 20:25:20.874300 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874167 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home" (OuterVolumeSpecName: "home") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:25:20.874404 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874302 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:25:20.874442 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874429 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:20.874486 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874441 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:20.874486 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.874449 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:25:20.876500 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.876478 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm" (OuterVolumeSpecName: "dshm") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:25:20.876660 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.876641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:25:20.876660 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.876650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt" (OuterVolumeSpecName: "kube-api-access-5hngt") pod "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" (UID: "e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429"). InnerVolumeSpecName "kube-api-access-5hngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:25:20.975212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.975123 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:20.975212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.975155 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:20.975212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.975165 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hngt\" (UniqueName: \"kubernetes.io/projected/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-kube-api-access-5hngt\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:20.975212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:20.975173 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:25:21.197541 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.197506 2573 generic.go:356] "Generic (PLEG): container finished" podID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerID="6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" exitCode=0 Sep 09 20:25:21.197737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.197564 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" Sep 09 20:25:21.197737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.197601 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" event={"ID":"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429","Type":"ContainerDied","Data":"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b"} Sep 09 20:25:21.197737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.197656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" event={"ID":"e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429","Type":"ContainerDied","Data":"43a882d616092ae7ee1e85f7a09d79a8cdb74024c49e18774349d5299c2b76e6"} Sep 09 20:25:21.197737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.197684 2573 scope.go:117] "RemoveContainer" containerID="6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" Sep 09 20:25:21.208390 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.208374 2573 scope.go:117] "RemoveContainer" containerID="6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" Sep 09 20:25:21.208691 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:25:21.208673 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b\": container with ID starting with 6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b not found: ID does not exist" containerID="6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b" Sep 09 20:25:21.208763 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.208704 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b"} err="failed to get container status \"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b\": rpc error: code = NotFound desc = could not find container \"6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b\": container with ID starting with 6d8e9d28cdcf8c632165e70819ee2e859ce1816b71315960f029ace97934447b not found: ID does not exist" Sep 09 20:25:21.221894 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.221869 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:25:21.226464 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.226407 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7"] Sep 09 20:25:21.701955 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:21.701910 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-747767b756-v8zh7" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 09 20:25:22.015804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:25:22.015727 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" path="/var/lib/kubelet/pods/e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429/volumes" Sep 09 20:26:43.581501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.581457 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:26:43.582124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.582104 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" Sep 09 20:26:43.582169 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.582129 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" Sep 09 20:26:43.582261 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.582248 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c3ea8c-d99e-4aaa-a60e-4f77a3fdf429" containerName="main" Sep 09 20:26:43.585850 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.585826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.589060 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.589032 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-22988\"" Sep 09 20:26:43.589198 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.589031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\"" Sep 09 20:26:43.589198 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.589096 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\"" Sep 09 20:26:43.589198 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.589095 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\"" Sep 09 20:26:43.593278 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.593251 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:26:43.615132 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.615347 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.615347 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwml7\" (UniqueName: \"kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.615476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.615476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.615476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.615439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716172 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716172 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716464 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716464 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716464 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwml7\" (UniqueName: \"kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716629 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716502 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716629 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716568 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716749 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716670 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.716860 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.716840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.718659 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.718641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.718917 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.718901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.724750 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.724715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwml7\" (UniqueName: \"kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:43.899485 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:43.899395 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:44.172623 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:44.172482 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:26:44.175034 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:26:44.175005 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5399212a_3cc8_4784_9742_a13e743225b4.slice/crio-1a00f70dc39cdb1ce7886b60c69202a4b42e1bca2437eadfe6fbd1eff70e4954 WatchSource:0}: Error finding container 1a00f70dc39cdb1ce7886b60c69202a4b42e1bca2437eadfe6fbd1eff70e4954: Status 404 returned error can't find the container with id 1a00f70dc39cdb1ce7886b60c69202a4b42e1bca2437eadfe6fbd1eff70e4954 Sep 09 20:26:44.616514 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:44.616475 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" event={"ID":"5399212a-3cc8-4784-9742-a13e743225b4","Type":"ContainerStarted","Data":"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11"} Sep 09 20:26:44.616904 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:44.616536 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" event={"ID":"5399212a-3cc8-4784-9742-a13e743225b4","Type":"ContainerStarted","Data":"1a00f70dc39cdb1ce7886b60c69202a4b42e1bca2437eadfe6fbd1eff70e4954"} Sep 09 20:26:44.634512 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:44.634456 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" podStartSLOduration=1.634440108 podStartE2EDuration="1.634440108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:44.632490344 +0000 UTC m=+1891.151370058" watchObservedRunningTime="2026-09-09 20:26:44.634440108 +0000 UTC m=+1891.153319821" Sep 09 20:26:44.899985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:44.899888 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:53.900191 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:53.900142 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:53.913362 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:53.913326 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:26:54.677105 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:26:54.677073 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:27:20.318338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.318303 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:27:20.328625 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:27:20.328586 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:27:20.328625 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:27:20.328625 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:27:20.328625 ip-10-0-141-196 kubenswrapper[2573]: > containerID="aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" cmd=["/bin/sleep","15"] Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.328647 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" message="" err=< Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:27:20.328660 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" containerID="cri-o://aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" Sep 09 20:27:20.328853 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.328725 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" containerID="cri-o://aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" gracePeriod=60 Sep 09 20:27:20.678024 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.677996 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:27:20.755498 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755550 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755579 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwml7\" (UniqueName: \"kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755628 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755651 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm\") pod \"5399212a-3cc8-4784-9742-a13e743225b4\" (UID: \"5399212a-3cc8-4784-9742-a13e743225b4\") " Sep 09 20:27:20.755926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755809 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:27:20.755926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755906 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home" (OuterVolumeSpecName: "home") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:27:20.755926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755917 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:27:20.756095 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.755921 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:20.758102 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.758079 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm" (OuterVolumeSpecName: "dshm") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:27:20.758183 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.758097 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:27:20.758183 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.758110 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7" (OuterVolumeSpecName: "kube-api-access-fwml7") pod "5399212a-3cc8-4784-9742-a13e743225b4" (UID: "5399212a-3cc8-4784-9742-a13e743225b4"). InnerVolumeSpecName "kube-api-access-fwml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:27:20.799646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.799606 2573 generic.go:356] "Generic (PLEG): container finished" podID="5399212a-3cc8-4784-9742-a13e743225b4" containerID="aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" exitCode=0 Sep 09 20:27:20.799840 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.799656 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" Sep 09 20:27:20.799840 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.799697 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" event={"ID":"5399212a-3cc8-4784-9742-a13e743225b4","Type":"ContainerDied","Data":"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11"} Sep 09 20:27:20.799840 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.799762 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" event={"ID":"5399212a-3cc8-4784-9742-a13e743225b4","Type":"ContainerDied","Data":"1a00f70dc39cdb1ce7886b60c69202a4b42e1bca2437eadfe6fbd1eff70e4954"} Sep 09 20:27:20.799840 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.799793 2573 scope.go:117] "RemoveContainer" containerID="aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" Sep 09 20:27:20.809595 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.809575 2573 scope.go:117] "RemoveContainer" containerID="aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" Sep 09 20:27:20.809897 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:27:20.809878 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11\": container with ID starting with aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11 not found: ID does not exist" containerID="aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11" Sep 09 20:27:20.809946 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.809907 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11"} err="failed to get container status \"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11\": rpc error: code = NotFound desc = could not find container \"aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11\": container with ID starting with aea04c3d94080820faeafdde087a0e2035e39d414c69823efbbe830bf7b08c11 not found: ID does not exist" Sep 09 20:27:20.820942 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.820913 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:27:20.824842 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.824818 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2"] Sep 09 20:27:20.856389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.856305 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5399212a-3cc8-4784-9742-a13e743225b4-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:20.856389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.856336 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwml7\" (UniqueName: \"kubernetes.io/projected/5399212a-3cc8-4784-9742-a13e743225b4-kube-api-access-fwml7\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:20.856389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.856346 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:20.856389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.856358 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:20.856389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:20.856366 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5399212a-3cc8-4784-9742-a13e743225b4-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:27:21.666577 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:21.666533 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b48cs2lk2" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded" Sep 09 20:27:22.018308 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:27:22.016680 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5399212a-3cc8-4784-9742-a13e743225b4" path="/var/lib/kubelet/pods/5399212a-3cc8-4784-9742-a13e743225b4/volumes" Sep 09 20:28:34.838359 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.838326 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:28:34.838814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.838733 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" Sep 09 20:28:34.838814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.838745 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" Sep 09 20:28:34.838890 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.838823 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5399212a-3cc8-4784-9742-a13e743225b4" containerName="main" Sep 09 20:28:34.842343 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.842321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.844664 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.844641 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:28:34.844894 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.844878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fcrwm\"" Sep 09 20:28:34.844936 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.844898 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\"" Sep 09 20:28:34.845547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.845525 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\"" Sep 09 20:28:34.845672 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.845601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\"" Sep 09 20:28:34.845744 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.845726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\"" Sep 09 20:28:34.852496 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.852469 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:28:34.917320 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98rf\" (UniqueName: \"kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:34.917554 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:34.917542 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019027 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.018987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019027 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019254 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019254 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98rf\" (UniqueName: \"kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019342 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019342 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019449 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019449 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019449 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019605 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019667 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.019767 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.019743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.020406 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.020384 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.021673 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.021653 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.021926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.021907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.026624 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.026600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98rf\" (UniqueName: \"kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.115940 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.115849 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:28:35.121073 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.121051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.142504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.142472 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:28:35.154506 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.154469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:35.221705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.221671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.221860 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.221748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.221860 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.221818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.221995 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.221921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323170 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323373 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323373 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323373 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323638 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323681 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.323899 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.323880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.325964 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.325942 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.432481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.432405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:28:35.445788 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.445500 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:28:35.447808 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:28:35.447778 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7551fb_b5d4_48bd_94b4_6159df8e8b89.slice/crio-99de0dfcea14a2d96bddbb45d894620b072f1716d8bcc5d6c885e740d097054d WatchSource:0}: Error finding container 99de0dfcea14a2d96bddbb45d894620b072f1716d8bcc5d6c885e740d097054d: Status 404 returned error can't find the container with id 99de0dfcea14a2d96bddbb45d894620b072f1716d8bcc5d6c885e740d097054d Sep 09 20:28:35.715655 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:35.715625 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:28:35.717042 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:28:35.717012 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6d9b5b_e20f_40c6_97e9_d99067691e13.slice/crio-30cd10ab64fe935600b434d21e310f31b960129b850a246af884fa68b53a74c5 WatchSource:0}: Error finding container 30cd10ab64fe935600b434d21e310f31b960129b850a246af884fa68b53a74c5: Status 404 returned error can't find the container with id 30cd10ab64fe935600b434d21e310f31b960129b850a246af884fa68b53a74c5 Sep 09 20:28:36.176710 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:36.176666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerStarted","Data":"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04"} Sep 09 20:28:36.177181 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:36.176735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerStarted","Data":"30cd10ab64fe935600b434d21e310f31b960129b850a246af884fa68b53a74c5"} Sep 09 20:28:36.178494 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:36.178467 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerStarted","Data":"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c"} Sep 09 20:28:36.178596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:36.178520 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerStarted","Data":"99de0dfcea14a2d96bddbb45d894620b072f1716d8bcc5d6c885e740d097054d"} Sep 09 20:28:39.198434 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:39.198394 2573 generic.go:356] "Generic (PLEG): container finished" podID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerID="376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04" exitCode=0 Sep 09 20:28:39.198877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:39.198472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerDied","Data":"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04"} Sep 09 20:28:39.199932 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:39.199915 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:28:39.200088 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:39.200064 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerID="f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c" exitCode=0 Sep 09 20:28:39.200159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:39.200137 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerDied","Data":"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c"} Sep 09 20:28:40.209870 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:40.209825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerStarted","Data":"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457"} Sep 09 20:28:40.232603 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:40.232537 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" podStartSLOduration=3.56832889 podStartE2EDuration="6.232513197s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.664184307s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:40.229199619 +0000 UTC m=+2006.748079334" watchObservedRunningTime="2026-09-09 20:28:40.232513197 +0000 UTC m=+2006.751392913" Sep 09 20:28:41.154881 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:41.154841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:45.155689 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:45.155563 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:45.173928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:45.173899 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:28:45.262941 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:28:45.262906 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:29:26.504326 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:29:26.504236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerStarted","Data":"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27"} Sep 09 20:29:26.504694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:29:26.504452 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:29:26.523453 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:29:26.523382 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" podStartSLOduration=1.933414996 podStartE2EDuration="51.523357598s" totalImagesPullingTime="47.054813929s" totalInitContainerRuntime="2.535128673s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:26.5209409 +0000 UTC m=+2053.039820616" watchObservedRunningTime="2026-09-09 20:29:26.523357598 +0000 UTC m=+2053.042237312" Sep 09 20:29:57.520659 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:29:57.520613 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:30:02.457285 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:02.457252 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc_4a6d9b5b-e20f-40c6-97e9-d99067691e13/main/0.log" Sep 09 20:30:38.141212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.141174 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:30:38.151588 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.151546 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:30:38.151588 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:30:38.151588 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:30:38.151588 ip-10-0-141-196 kubenswrapper[2573]: > containerID="4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" cmd=["/bin/sleep","15"] Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.151609 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" message="" err=< Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.151623 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" containerID="cri-o://4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" Sep 09 20:30:38.151945 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.151674 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" containerID="cri-o://4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" gracePeriod=60 Sep 09 20:30:38.159542 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.159508 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:30:38.159769 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.159730 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="main" containerID="cri-o://86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27" gracePeriod=30 Sep 09 20:30:38.245915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.245877 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Sep 09 20:30:38.502865 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.502840 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:30:38.538998 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.538963 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539012 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539028 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539065 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539109 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539135 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l98rf\" (UniqueName: \"kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539200 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539241 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home\") pod \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\" (UID: \"1a7551fb-b5d4-48bd-94b4-6159df8e8b89\") " Sep 09 20:30:38.539487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539286 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.539487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539355 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.539487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539414 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home" (OuterVolumeSpecName: "home") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.539758 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539616 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.539758 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539635 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.539758 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539651 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.539911 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.539748 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:30:38.542504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.542420 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm" (OuterVolumeSpecName: "dshm") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.542504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.542421 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:30:38.542504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.542463 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf" (OuterVolumeSpecName: "kube-api-access-l98rf") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "kube-api-access-l98rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:30:38.561430 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.561386 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a7551fb-b5d4-48bd-94b4-6159df8e8b89" (UID: "1a7551fb-b5d4-48bd-94b4-6159df8e8b89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.640412 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.640382 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.640412 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.640413 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l98rf\" (UniqueName: \"kubernetes.io/projected/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kube-api-access-l98rf\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.640595 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.640430 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.640595 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.640443 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.640595 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.640454 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7551fb-b5d4-48bd-94b4-6159df8e8b89-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.793648 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.793624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:30:38.842139 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842107 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir\") pod \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " Sep 09 20:30:38.842371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842149 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert\") pod \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " Sep 09 20:30:38.842371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842183 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs\") pod \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " Sep 09 20:30:38.842371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842250 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location\") pod \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\" (UID: \"4a6d9b5b-e20f-40c6-97e9-d99067691e13\") " Sep 09 20:30:38.842535 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842510 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a6d9b5b-e20f-40c6-97e9-d99067691e13" (UID: "4a6d9b5b-e20f-40c6-97e9-d99067691e13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:30:38.842588 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.842567 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a6d9b5b-e20f-40c6-97e9-d99067691e13" (UID: "4a6d9b5b-e20f-40c6-97e9-d99067691e13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.844501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.844481 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a6d9b5b-e20f-40c6-97e9-d99067691e13" (UID: "4a6d9b5b-e20f-40c6-97e9-d99067691e13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:30:38.864656 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.864601 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a6d9b5b-e20f-40c6-97e9-d99067691e13" (UID: "4a6d9b5b-e20f-40c6-97e9-d99067691e13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:30:38.874801 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.874770 2573 generic.go:356] "Generic (PLEG): container finished" podID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerID="86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27" exitCode=0 Sep 09 20:30:38.874936 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.874838 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" Sep 09 20:30:38.874936 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.874859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerDied","Data":"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27"} Sep 09 20:30:38.874936 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.874925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc" event={"ID":"4a6d9b5b-e20f-40c6-97e9-d99067691e13","Type":"ContainerDied","Data":"30cd10ab64fe935600b434d21e310f31b960129b850a246af884fa68b53a74c5"} Sep 09 20:30:38.875106 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.874954 2573 scope.go:117] "RemoveContainer" containerID="86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27" Sep 09 20:30:38.876777 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.876754 2573 generic.go:356] "Generic (PLEG): container finished" podID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerID="4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" exitCode=0 Sep 09 20:30:38.876889 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.876839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerDied","Data":"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457"} Sep 09 20:30:38.876889 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.876859 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" Sep 09 20:30:38.876889 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.876878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5" event={"ID":"1a7551fb-b5d4-48bd-94b4-6159df8e8b89","Type":"ContainerDied","Data":"99de0dfcea14a2d96bddbb45d894620b072f1716d8bcc5d6c885e740d097054d"} Sep 09 20:30:38.885931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.885910 2573 scope.go:117] "RemoveContainer" containerID="376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04" Sep 09 20:30:38.894296 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.894278 2573 scope.go:117] "RemoveContainer" containerID="86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27" Sep 09 20:30:38.894597 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.894578 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27\": container with ID starting with 86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27 not found: ID does not exist" containerID="86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27" Sep 09 20:30:38.894653 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.894607 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27"} err="failed to get container status \"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27\": rpc error: code = NotFound desc = could not find container \"86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27\": container with ID starting with 86347e518a247788a9c15e5ff0525900278d65b5b65061c2c42a31eead3e2a27 not found: ID does not exist" Sep 09 20:30:38.894653 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.894627 2573 scope.go:117] "RemoveContainer" containerID="376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04" Sep 09 20:30:38.894865 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.894851 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04\": container with ID starting with 376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04 not found: ID does not exist" containerID="376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04" Sep 09 20:30:38.894912 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.894868 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04"} err="failed to get container status \"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04\": rpc error: code = NotFound desc = could not find container \"376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04\": container with ID starting with 376310dc8fcc4f025d7ea9dc2dc3d03322db76e6db860530e658487bfb212f04 not found: ID does not exist" Sep 09 20:30:38.894912 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.894880 2573 scope.go:117] "RemoveContainer" containerID="4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" Sep 09 20:30:38.899594 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.899570 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:30:38.903856 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.903829 2573 scope.go:117] "RemoveContainer" containerID="f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c" Sep 09 20:30:38.906073 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.906050 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598bcf44b7-nk7gc"] Sep 09 20:30:38.912386 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.912365 2573 scope.go:117] "RemoveContainer" containerID="4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" Sep 09 20:30:38.912693 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.912673 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457\": container with ID starting with 4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457 not found: ID does not exist" containerID="4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457" Sep 09 20:30:38.912760 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.912700 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457"} err="failed to get container status \"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457\": rpc error: code = NotFound desc = could not find container \"4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457\": container with ID starting with 4fa0b455f17b557e18e15343ca3d1e0fcc64e0ed755151bec224e04f5b902457 not found: ID does not exist" Sep 09 20:30:38.912760 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.912721 2573 scope.go:117] "RemoveContainer" containerID="f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c" Sep 09 20:30:38.913011 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:30:38.912993 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c\": container with ID starting with f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c not found: ID does not exist" containerID="f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c" Sep 09 20:30:38.913053 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.913019 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c"} err="failed to get container status \"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c\": rpc error: code = NotFound desc = could not find container \"f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c\": container with ID starting with f816a298df33d1716f760a8887d88138107d9010258fc2265ce13e67a843304c not found: ID does not exist" Sep 09 20:30:38.918803 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.918782 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:30:38.920639 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.920617 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-r5ct5"] Sep 09 20:30:38.943874 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.943804 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.943874 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.943828 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a6d9b5b-e20f-40c6-97e9-d99067691e13-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.943874 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.943836 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6d9b5b-e20f-40c6-97e9-d99067691e13-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:38.943874 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:38.943845 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a6d9b5b-e20f-40c6-97e9-d99067691e13-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:30:40.015073 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:40.015035 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" path="/var/lib/kubelet/pods/1a7551fb-b5d4-48bd-94b4-6159df8e8b89/volumes" Sep 09 20:30:40.015490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:30:40.015482 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" path="/var/lib/kubelet/pods/4a6d9b5b-e20f-40c6-97e9-d99067691e13/volumes" Sep 09 20:31:33.022020 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.021984 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:31:33.022688 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022665 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="storage-initializer" Sep 09 20:31:33.022753 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022703 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="storage-initializer" Sep 09 20:31:33.022753 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022719 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="storage-initializer" Sep 09 20:31:33.022753 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022728 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="storage-initializer" Sep 09 20:31:33.022753 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022747 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" Sep 09 20:31:33.022877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022755 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" Sep 09 20:31:33.022877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022773 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="main" Sep 09 20:31:33.022877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022782 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="main" Sep 09 20:31:33.022877 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022869 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7551fb-b5d4-48bd-94b4-6159df8e8b89" containerName="main" Sep 09 20:31:33.023000 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.022886 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6d9b5b-e20f-40c6-97e9-d99067691e13" containerName="main" Sep 09 20:31:33.026804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.026783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.029045 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029023 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\"" Sep 09 20:31:33.029689 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029651 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\"" Sep 09 20:31:33.029689 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029668 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\"" Sep 09 20:31:33.029689 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029661 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:31:33.029920 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029711 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-n74bf\"" Sep 09 20:31:33.029920 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.029661 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\"" Sep 09 20:31:33.037327 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.037300 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:31:33.145421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145421 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145691 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145691 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j87\" (UniqueName: \"kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145691 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.145809 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.145734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.246886 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j87\" (UniqueName: \"kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.246989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247430 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247595 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247676 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.247821 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.247804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.249701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.249683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.249783 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.249767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.255820 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.255797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j87\" (UniqueName: \"kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-n92zx\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.329424 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.329396 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:31:33.334390 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.334369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.340219 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.340196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:33.344944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.344918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:31:33.448498 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.448458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.448653 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.448594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.448653 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.448631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.448758 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.448655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550045 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550264 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550319 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550319 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550473 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550456 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.550902 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.550883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.552842 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.552820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.628331 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.628306 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:31:33.630512 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:31:33.630488 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc59aa14_37e1_4a1f_993d_a9008bdbda75.slice/crio-67dddda63dc42c700e534c1893140c35d0789d0ec32daa678d8715885f0fd29b WatchSource:0}: Error finding container 67dddda63dc42c700e534c1893140c35d0789d0ec32daa678d8715885f0fd29b: Status 404 returned error can't find the container with id 67dddda63dc42c700e534c1893140c35d0789d0ec32daa678d8715885f0fd29b Sep 09 20:31:33.648654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.648634 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:33.922569 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:33.922485 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:31:33.925604 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:31:33.925579 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f49a63d_fb7a_4831_92f0_2b8a17ee4494.slice/crio-863a3e9f684cb68fb5d391d9454ba9ccea737a49e664ccfdf170047234b796f1 WatchSource:0}: Error finding container 863a3e9f684cb68fb5d391d9454ba9ccea737a49e664ccfdf170047234b796f1: Status 404 returned error can't find the container with id 863a3e9f684cb68fb5d391d9454ba9ccea737a49e664ccfdf170047234b796f1 Sep 09 20:31:34.151587 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:34.151544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerStarted","Data":"40ed48b5ffa8930b9c3c158e6764ec7e8741df83e332f960295bf48164580c82"} Sep 09 20:31:34.152070 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:34.151621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerStarted","Data":"67dddda63dc42c700e534c1893140c35d0789d0ec32daa678d8715885f0fd29b"} Sep 09 20:31:34.153293 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:34.153260 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerStarted","Data":"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925"} Sep 09 20:31:34.153495 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:34.153308 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerStarted","Data":"863a3e9f684cb68fb5d391d9454ba9ccea737a49e664ccfdf170047234b796f1"} Sep 09 20:31:37.170966 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:37.170923 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerID="40ed48b5ffa8930b9c3c158e6764ec7e8741df83e332f960295bf48164580c82" exitCode=0 Sep 09 20:31:37.171357 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:37.170989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerDied","Data":"40ed48b5ffa8930b9c3c158e6764ec7e8741df83e332f960295bf48164580c82"} Sep 09 20:31:37.172772 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:37.172748 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerID="8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925" exitCode=0 Sep 09 20:31:37.172879 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:37.172825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerDied","Data":"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925"} Sep 09 20:31:38.180622 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.180583 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerStarted","Data":"245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc"} Sep 09 20:31:38.182783 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.182759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerStarted","Data":"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c"} Sep 09 20:31:38.182989 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.182971 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:31:38.202369 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.202315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" podStartSLOduration=2.695031344 podStartE2EDuration="5.202297224s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.50726588s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:38.200352618 +0000 UTC m=+2184.719232328" watchObservedRunningTime="2026-09-09 20:31:38.202297224 +0000 UTC m=+2184.721176971" Sep 09 20:31:38.217652 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.217602 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" podStartSLOduration=2.709800263 podStartE2EDuration="5.217584942s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.507784679s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:38.215650409 +0000 UTC m=+2184.734530126" watchObservedRunningTime="2026-09-09 20:31:38.217584942 +0000 UTC m=+2184.736464655" Sep 09 20:31:38.341116 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:38.341079 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:43.340645 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:43.340604 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:43.353459 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:43.353430 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:31:44.227367 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:31:44.227338 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:32:09.200073 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:09.200042 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:32:25.335254 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:25.335202 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn_6f49a63d-fb7a-4831-92f0-2b8a17ee4494/main/0.log" Sep 09 20:32:29.189523 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.189490 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:32:29.190113 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.189710 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="main" containerID="cri-o://f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c" gracePeriod=30 Sep 09 20:32:29.197084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.197051 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:32:29.207167 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:32:29.207128 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:32:29.207167 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:32:29.207167 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:32:29.207167 ip-10-0-141-196 kubenswrapper[2573]: > containerID="245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc" cmd=["/bin/sleep","15"] Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.207183 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" message="" err=< Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:32:29.207196 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" containerID="cri-o://245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc" Sep 09 20:32:29.207469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.207259 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" containerID="cri-o://245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc" gracePeriod=60 Sep 09 20:32:29.215236 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.215180 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Sep 09 20:32:29.451955 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.451860 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerID="245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc" exitCode=0 Sep 09 20:32:29.451955 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.451934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerDied","Data":"245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc"} Sep 09 20:32:29.565130 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.565101 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:32:29.683339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683307 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683349 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683405 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683432 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4j87\" (UniqueName: \"kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683465 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683501 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683596 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm\") pod \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\" (UID: \"fc59aa14-37e1-4a1f-993d-a9008bdbda75\") " Sep 09 20:32:29.683818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683653 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home" (OuterVolumeSpecName: "home") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.683818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683678 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.683818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683729 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:32:29.684066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.684066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683958 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.684066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683977 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.684066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.683990 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc59aa14-37e1-4a1f-993d-a9008bdbda75-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.684066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.684003 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.685943 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.685903 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:32:29.686055 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.685945 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87" (OuterVolumeSpecName: "kube-api-access-c4j87") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "kube-api-access-c4j87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:32:29.686055 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.685979 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm" (OuterVolumeSpecName: "dshm") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.710890 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.710853 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc59aa14-37e1-4a1f-993d-a9008bdbda75" (UID: "fc59aa14-37e1-4a1f-993d-a9008bdbda75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.784627 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.784592 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4j87\" (UniqueName: \"kubernetes.io/projected/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kube-api-access-c4j87\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.784627 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.784630 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.784854 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.784645 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc59aa14-37e1-4a1f-993d-a9008bdbda75-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.784854 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.784659 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc59aa14-37e1-4a1f-993d-a9008bdbda75-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:29.832498 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.832467 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:32:29.986312 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986194 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs\") pod \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " Sep 09 20:32:29.986312 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986311 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert\") pod \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " Sep 09 20:32:29.986561 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986354 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir\") pod \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " Sep 09 20:32:29.986561 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986395 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location\") pod \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\" (UID: \"6f49a63d-fb7a-4831-92f0-2b8a17ee4494\") " Sep 09 20:32:29.986722 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986698 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f49a63d-fb7a-4831-92f0-2b8a17ee4494" (UID: "6f49a63d-fb7a-4831-92f0-2b8a17ee4494"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:32:29.986787 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.986759 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f49a63d-fb7a-4831-92f0-2b8a17ee4494" (UID: "6f49a63d-fb7a-4831-92f0-2b8a17ee4494"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:29.988620 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:29.988597 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f49a63d-fb7a-4831-92f0-2b8a17ee4494" (UID: "6f49a63d-fb7a-4831-92f0-2b8a17ee4494"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:32:30.009080 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.009034 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f49a63d-fb7a-4831-92f0-2b8a17ee4494" (UID: "6f49a63d-fb7a-4831-92f0-2b8a17ee4494"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:32:30.088041 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.088004 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:30.088041 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.088051 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:30.088285 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.088060 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:30.088285 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.088077 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f49a63d-fb7a-4831-92f0-2b8a17ee4494-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:32:30.458532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.458501 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" Sep 09 20:32:30.458532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.458525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx" event={"ID":"fc59aa14-37e1-4a1f-993d-a9008bdbda75","Type":"ContainerDied","Data":"67dddda63dc42c700e534c1893140c35d0789d0ec32daa678d8715885f0fd29b"} Sep 09 20:32:30.459080 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.458595 2573 scope.go:117] "RemoveContainer" containerID="245d0bf79bbaf931f7632546688c68dc19d5e9a2a548e228bcb74f3fd4b48fbc" Sep 09 20:32:30.460362 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.460333 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerID="f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c" exitCode=0 Sep 09 20:32:30.460504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.460402 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" Sep 09 20:32:30.460694 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.460422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerDied","Data":"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c"} Sep 09 20:32:30.460825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.460729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn" event={"ID":"6f49a63d-fb7a-4831-92f0-2b8a17ee4494","Type":"ContainerDied","Data":"863a3e9f684cb68fb5d391d9454ba9ccea737a49e664ccfdf170047234b796f1"} Sep 09 20:32:30.468403 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.468384 2573 scope.go:117] "RemoveContainer" containerID="40ed48b5ffa8930b9c3c158e6764ec7e8741df83e332f960295bf48164580c82" Sep 09 20:32:30.476186 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.476155 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:32:30.477596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.477565 2573 scope.go:117] "RemoveContainer" containerID="f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c" Sep 09 20:32:30.479250 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.479209 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-n92zx"] Sep 09 20:32:30.485887 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.485866 2573 scope.go:117] "RemoveContainer" containerID="8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925" Sep 09 20:32:30.489410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.489384 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:32:30.495613 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.495590 2573 scope.go:117] "RemoveContainer" containerID="f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c" Sep 09 20:32:30.495715 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.495676 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c5f777fb9-s7bqn"] Sep 09 20:32:30.495946 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:32:30.495928 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c\": container with ID starting with f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c not found: ID does not exist" containerID="f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c" Sep 09 20:32:30.495996 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.495957 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c"} err="failed to get container status \"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c\": rpc error: code = NotFound desc = could not find container \"f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c\": container with ID starting with f8ca8feeb0ba52d012b2d1b9a2423571fcfa65db35aa514d29650cee8209da2c not found: ID does not exist" Sep 09 20:32:30.495996 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.495976 2573 scope.go:117] "RemoveContainer" containerID="8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925" Sep 09 20:32:30.496276 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:32:30.496259 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925\": container with ID starting with 8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925 not found: ID does not exist" containerID="8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925" Sep 09 20:32:30.496332 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:30.496281 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925"} err="failed to get container status \"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925\": rpc error: code = NotFound desc = could not find container \"8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925\": container with ID starting with 8ccef88bab65cd2d8fcdf189d149a0f379a3f45cf23999bde7c79bf94883b925 not found: ID does not exist" Sep 09 20:32:32.015403 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:32.015367 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" path="/var/lib/kubelet/pods/6f49a63d-fb7a-4831-92f0-2b8a17ee4494/volumes" Sep 09 20:32:32.015823 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:32:32.015809 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" path="/var/lib/kubelet/pods/fc59aa14-37e1-4a1f-993d-a9008bdbda75/volumes" Sep 09 20:33:22.368197 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368154 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:33:22.368745 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368719 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="main" Sep 09 20:33:22.368745 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368735 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="main" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368753 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="storage-initializer" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368759 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="storage-initializer" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368767 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368773 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368780 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="storage-initializer" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368786 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="storage-initializer" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368840 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f49a63d-fb7a-4831-92f0-2b8a17ee4494" containerName="main" Sep 09 20:33:22.368862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.368851 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc59aa14-37e1-4a1f-993d-a9008bdbda75" containerName="main" Sep 09 20:33:22.372111 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.372081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.375422 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.375400 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\"" Sep 09 20:33:22.375559 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.375437 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:33:22.375559 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.375449 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\"" Sep 09 20:33:22.376148 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.376126 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\"" Sep 09 20:33:22.376293 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.376147 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-89pnb\"" Sep 09 20:33:22.376293 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.376238 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\"" Sep 09 20:33:22.384837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.384799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:33:22.478059 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478180 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q67\" (UniqueName: \"kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.478309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.478306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.579631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.579825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.579825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.579825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.579825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579881 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q67\" (UniqueName: \"kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.579911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.580021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580283 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.580061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580459 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.580433 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580564 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.580544 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.580706 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.580685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.582177 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.582158 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.582535 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.582518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.591410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.591388 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q67\" (UniqueName: \"kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67\") pod \"tokenizer-template-args-test-kserve-5774447cc5-zpmh8\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.619944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.619851 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:33:22.623869 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.623845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.636432 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.636402 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:33:22.683737 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.683700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:22.781381 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.781339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.781381 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.781387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.781592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.781479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.781592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.781549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882424 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882424 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882678 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882823 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.882883 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.882800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.883099 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.883082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.885345 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.885321 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-679fcff497-7vz6l\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.939409 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.939377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:22.960907 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:22.960884 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:33:22.963367 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:33:22.963339 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca96f0c_ec6d_4d7a_be20_9d43e70f605c.slice/crio-7dfde1242734c927e6135705511143f8ccaad4ebfea2ebb323f415133116256c WatchSource:0}: Error finding container 7dfde1242734c927e6135705511143f8ccaad4ebfea2ebb323f415133116256c: Status 404 returned error can't find the container with id 7dfde1242734c927e6135705511143f8ccaad4ebfea2ebb323f415133116256c Sep 09 20:33:23.216375 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:23.216340 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:33:23.218943 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:33:23.218917 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be1197a_1a28_4d41_8b1a_1114792dbdb2.slice/crio-d0d71a691174d371cc8fe0d50fda9499b56b5156cee26e95fb974e01cb7f8b4c WatchSource:0}: Error finding container d0d71a691174d371cc8fe0d50fda9499b56b5156cee26e95fb974e01cb7f8b4c: Status 404 returned error can't find the container with id d0d71a691174d371cc8fe0d50fda9499b56b5156cee26e95fb974e01cb7f8b4c Sep 09 20:33:23.736814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:23.736772 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerStarted","Data":"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9"} Sep 09 20:33:23.737302 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:23.736837 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerStarted","Data":"d0d71a691174d371cc8fe0d50fda9499b56b5156cee26e95fb974e01cb7f8b4c"} Sep 09 20:33:23.738602 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:23.738562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerStarted","Data":"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803"} Sep 09 20:33:23.738602 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:23.738602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerStarted","Data":"7dfde1242734c927e6135705511143f8ccaad4ebfea2ebb323f415133116256c"} Sep 09 20:33:25.751724 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:25.751674 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerID="1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803" exitCode=0 Sep 09 20:33:25.752086 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:25.751749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerDied","Data":"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803"} Sep 09 20:33:26.758415 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:26.758378 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerStarted","Data":"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb"} Sep 09 20:33:26.759945 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:26.759919 2573 generic.go:356] "Generic (PLEG): container finished" podID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerID="ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9" exitCode=0 Sep 09 20:33:26.760063 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:26.759981 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerDied","Data":"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9"} Sep 09 20:33:26.781275 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:26.781193 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" podStartSLOduration=2.2730026629999998 podStartE2EDuration="4.781174705s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.508172042s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:26.779144126 +0000 UTC m=+2293.298023838" watchObservedRunningTime="2026-09-09 20:33:26.781174705 +0000 UTC m=+2293.300054415" Sep 09 20:33:27.684498 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:27.684466 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:27.767315 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:27.767280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerStarted","Data":"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59"} Sep 09 20:33:27.767785 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:27.767640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:33:27.784371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:27.784317 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" podStartSLOduration=3.160400141 podStartE2EDuration="5.78430078s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.623900639s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:27.782989003 +0000 UTC m=+2294.301868719" watchObservedRunningTime="2026-09-09 20:33:27.78430078 +0000 UTC m=+2294.303180493" Sep 09 20:33:32.684172 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:32.684135 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:32.697358 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:32.697323 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:32.806267 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:32.806220 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:33:58.785457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:33:58.785416 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:34:10.266735 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:10.266653 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-679fcff497-7vz6l_8be1197a-1a28-4d41-8b1a-1114792dbdb2/main/0.log" Sep 09 20:34:12.874576 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:12.874541 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:34:12.874986 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:12.874755 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="main" containerID="cri-o://f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59" gracePeriod=30 Sep 09 20:34:12.891115 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:12.891073 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:34:12.901054 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:12.901012 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:34:12.901054 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:34:12.901054 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:34:12.901054 ip-10-0-141-196 kubenswrapper[2573]: > containerID="a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" cmd=["/bin/sleep","15"] Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:12.901072 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" message="" err=< Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:12.901085 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="main" containerID="cri-o://a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" Sep 09 20:34:12.901372 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:12.901142 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="main" containerID="cri-o://a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" gracePeriod=60 Sep 09 20:34:13.262651 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.262624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:34:13.368636 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368596 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.368833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368650 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.368833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368672 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.368833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368711 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.368833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368778 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.369066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368916 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2q67\" (UniqueName: \"kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.369066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368936 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.369066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368960 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.369066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.368971 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home" (OuterVolumeSpecName: "home") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.369066 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369000 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache\") pod \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\" (UID: \"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c\") " Sep 09 20:34:13.369348 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369085 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:34:13.369348 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369334 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.369457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369380 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.369457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369400 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.369457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.369413 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.371487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.371451 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:34:13.371603 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.371520 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm" (OuterVolumeSpecName: "dshm") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.371603 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.371563 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67" (OuterVolumeSpecName: "kube-api-access-c2q67") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "kube-api-access-c2q67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:34:13.400335 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.400296 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" (UID: "9ca96f0c-ec6d-4d7a-be20-9d43e70f605c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.470789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.470685 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.470789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.470722 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.470789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.470739 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q67\" (UniqueName: \"kubernetes.io/projected/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-kube-api-access-c2q67\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.470789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.470753 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.470789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.470765 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.546975 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.546949 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:34:13.571504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571472 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir\") pod \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " Sep 09 20:34:13.571504 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert\") pod \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " Sep 09 20:34:13.571746 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571546 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs\") pod \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " Sep 09 20:34:13.571746 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571622 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location\") pod \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\" (UID: \"8be1197a-1a28-4d41-8b1a-1114792dbdb2\") " Sep 09 20:34:13.571966 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571863 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8be1197a-1a28-4d41-8b1a-1114792dbdb2" (UID: "8be1197a-1a28-4d41-8b1a-1114792dbdb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.571966 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.571952 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8be1197a-1a28-4d41-8b1a-1114792dbdb2" (UID: "8be1197a-1a28-4d41-8b1a-1114792dbdb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:34:13.574108 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.574081 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8be1197a-1a28-4d41-8b1a-1114792dbdb2" (UID: "8be1197a-1a28-4d41-8b1a-1114792dbdb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:34:13.597844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.597791 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8be1197a-1a28-4d41-8b1a-1114792dbdb2" (UID: "8be1197a-1a28-4d41-8b1a-1114792dbdb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:34:13.672602 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.672555 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.672602 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.672597 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.672602 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.672612 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8be1197a-1a28-4d41-8b1a-1114792dbdb2-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:13.672854 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:13.672625 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1197a-1a28-4d41-8b1a-1114792dbdb2-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:34:14.008279 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.008209 2573 generic.go:356] "Generic (PLEG): container finished" podID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerID="f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59" exitCode=0 Sep 09 20:34:14.008279 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.008267 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerDied","Data":"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59"} Sep 09 20:34:14.008786 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.008288 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" Sep 09 20:34:14.008786 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.008351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l" event={"ID":"8be1197a-1a28-4d41-8b1a-1114792dbdb2","Type":"ContainerDied","Data":"d0d71a691174d371cc8fe0d50fda9499b56b5156cee26e95fb974e01cb7f8b4c"} Sep 09 20:34:14.008786 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.008379 2573 scope.go:117] "RemoveContainer" containerID="f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59" Sep 09 20:34:14.010731 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.010708 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerID="a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" exitCode=0 Sep 09 20:34:14.012280 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.012257 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" Sep 09 20:34:14.018209 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.018174 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerDied","Data":"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb"} Sep 09 20:34:14.018472 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.018444 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8" event={"ID":"9ca96f0c-ec6d-4d7a-be20-9d43e70f605c","Type":"ContainerDied","Data":"7dfde1242734c927e6135705511143f8ccaad4ebfea2ebb323f415133116256c"} Sep 09 20:34:14.026974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.026943 2573 scope.go:117] "RemoveContainer" containerID="ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9" Sep 09 20:34:14.036533 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.036512 2573 scope.go:117] "RemoveContainer" containerID="f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59" Sep 09 20:34:14.036883 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:14.036866 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59\": container with ID starting with f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59 not found: ID does not exist" containerID="f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59" Sep 09 20:34:14.036928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.036892 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59"} err="failed to get container status \"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59\": rpc error: code = NotFound desc = could not find container \"f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59\": container with ID starting with f0fd98a93a9d1fe74dff62db7b8adc2887a85050d76870ee297275af4e589b59 not found: ID does not exist" Sep 09 20:34:14.036928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.036910 2573 scope.go:117] "RemoveContainer" containerID="ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9" Sep 09 20:34:14.037241 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:14.037203 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9\": container with ID starting with ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9 not found: ID does not exist" containerID="ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9" Sep 09 20:34:14.037407 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.037329 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9"} err="failed to get container status \"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9\": rpc error: code = NotFound desc = could not find container \"ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9\": container with ID starting with ec593390b016ac18e83e6f34d8977cfa2c9c23fc8f65134a34af1c9d6312c7f9 not found: ID does not exist" Sep 09 20:34:14.037407 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.037348 2573 scope.go:117] "RemoveContainer" containerID="a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" Sep 09 20:34:14.046896 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.046873 2573 scope.go:117] "RemoveContainer" containerID="1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803" Sep 09 20:34:14.048808 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.048787 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:34:14.054026 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.053999 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-zpmh8"] Sep 09 20:34:14.056574 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.056551 2573 scope.go:117] "RemoveContainer" containerID="a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" Sep 09 20:34:14.056942 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:14.056919 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb\": container with ID starting with a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb not found: ID does not exist" containerID="a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb" Sep 09 20:34:14.056993 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.056952 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb"} err="failed to get container status \"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb\": rpc error: code = NotFound desc = could not find container \"a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb\": container with ID starting with a28602e58e2880ea90871a692260aa5ecb21cdee83087e9b59f704b10128c0fb not found: ID does not exist" Sep 09 20:34:14.056993 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.056973 2573 scope.go:117] "RemoveContainer" containerID="1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803" Sep 09 20:34:14.057286 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:34:14.057268 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803\": container with ID starting with 1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803 not found: ID does not exist" containerID="1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803" Sep 09 20:34:14.057342 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.057289 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803"} err="failed to get container status \"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803\": rpc error: code = NotFound desc = could not find container \"1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803\": container with ID starting with 1d188c7e60ce2eda990c45f14bb792321037c38096b576c7ab50b267ccb7d803 not found: ID does not exist" Sep 09 20:34:14.075382 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.075348 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:34:14.085837 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:14.085800 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-679fcff497-7vz6l"] Sep 09 20:34:16.021071 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:16.021032 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" path="/var/lib/kubelet/pods/8be1197a-1a28-4d41-8b1a-1114792dbdb2/volumes" Sep 09 20:34:16.021568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:34:16.021539 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" path="/var/lib/kubelet/pods/9ca96f0c-ec6d-4d7a-be20-9d43e70f605c/volumes" Sep 09 20:37:23.923530 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923492 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923889 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="main" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923900 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="main" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923914 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="storage-initializer" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923920 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="storage-initializer" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923932 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="storage-initializer" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923938 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="storage-initializer" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923948 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="main" Sep 09 20:37:23.923956 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.923952 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="main" Sep 09 20:37:23.924256 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.924004 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca96f0c-ec6d-4d7a-be20-9d43e70f605c" containerName="main" Sep 09 20:37:23.924256 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.924014 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be1197a-1a28-4d41-8b1a-1114792dbdb2" containerName="main" Sep 09 20:37:23.927261 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.927243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.930308 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.930269 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\"" Sep 09 20:37:23.930475 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.930323 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\"" Sep 09 20:37:23.930475 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.930364 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\"" Sep 09 20:37:23.930475 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.930378 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l6wdn\"" Sep 09 20:37:23.935835 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.935810 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:37:23.998604 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.998799 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nd8\" (UniqueName: \"kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.998799 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.998911 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.998911 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:23.998989 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:23.998927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.099641 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.099838 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.099838 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.099838 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.100008 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nd8\" (UniqueName: \"kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.100008 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.100008 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.099988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.100194 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.100028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.100328 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.100304 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.102314 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.102296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.102705 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.102684 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.108096 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.108059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nd8\" (UniqueName: \"kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.238886 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.238792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:24.514272 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.514243 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:37:24.516878 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:37:24.516845 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d7898a_a2ae_45bf_9613_34429dc14823.slice/crio-b1d4d1cec2b2264e0a23a27d38402f8a481626695563cbafad067918177ba67f WatchSource:0}: Error finding container b1d4d1cec2b2264e0a23a27d38402f8a481626695563cbafad067918177ba67f: Status 404 returned error can't find the container with id b1d4d1cec2b2264e0a23a27d38402f8a481626695563cbafad067918177ba67f Sep 09 20:37:24.963021 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.962983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" event={"ID":"98d7898a-a2ae-45bf-9613-34429dc14823","Type":"ContainerStarted","Data":"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677"} Sep 09 20:37:24.963534 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.963036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" event={"ID":"98d7898a-a2ae-45bf-9613-34429dc14823","Type":"ContainerStarted","Data":"b1d4d1cec2b2264e0a23a27d38402f8a481626695563cbafad067918177ba67f"} Sep 09 20:37:24.984776 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:24.984725 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" podStartSLOduration=1.984708731 podStartE2EDuration="1.984708731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:24.983314253 +0000 UTC m=+2531.502193968" watchObservedRunningTime="2026-09-09 20:37:24.984708731 +0000 UTC m=+2531.503588443" Sep 09 20:37:25.239677 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:25.239581 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:34.239569 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:34.239529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:34.252129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:34.252104 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:37:35.022287 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:37:35.022252 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:40:04.334959 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:04.334918 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:40:04.768476 2573 log.go:32] "ExecSync cmd from runtime service failed" err=< Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: > containerID="eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" cmd=["/bin/sleep","15"] Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:04.768539 2573 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" message="" err=< Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: > Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:40:04.768555 2573 kuberuntime_container.go:755] "PreStop hook failed" err=< Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: , stdout: , stderr: , exit code -1 Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" containerID="cri-o://eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" Sep 09 20:40:04.768607 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:04.768601 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" containerID="cri-o://eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" gracePeriod=60 Sep 09 20:40:05.011663 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.011620 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Sep 09 20:40:05.115328 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.115302 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:40:05.180179 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180138 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180179 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180180 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180456 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180274 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180456 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180338 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180456 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180375 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88nd8\" (UniqueName: \"kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180456 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180428 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home\") pod \"98d7898a-a2ae-45bf-9613-34429dc14823\" (UID: \"98d7898a-a2ae-45bf-9613-34429dc14823\") " Sep 09 20:40:05.180671 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180456 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:40:05.180671 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180469 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache" (OuterVolumeSpecName: "model-cache") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:40:05.180775 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180717 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.180775 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180749 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home" (OuterVolumeSpecName: "home") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:40:05.180775 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.180759 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.182961 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.182915 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:40:05.182961 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.182929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8" (OuterVolumeSpecName: "kube-api-access-88nd8") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "kube-api-access-88nd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:40:05.182961 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.182928 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm" (OuterVolumeSpecName: "dshm") pod "98d7898a-a2ae-45bf-9613-34429dc14823" (UID: "98d7898a-a2ae-45bf-9613-34429dc14823"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:40:05.281566 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.281532 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d7898a-a2ae-45bf-9613-34429dc14823-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.281566 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.281563 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.281566 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.281572 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88nd8\" (UniqueName: \"kubernetes.io/projected/98d7898a-a2ae-45bf-9613-34429dc14823-kube-api-access-88nd8\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.281825 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.281582 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d7898a-a2ae-45bf-9613-34429dc14823-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:40:05.764459 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.764427 2573 generic.go:356] "Generic (PLEG): container finished" podID="98d7898a-a2ae-45bf-9613-34429dc14823" containerID="eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" exitCode=0 Sep 09 20:40:05.764924 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.764473 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" Sep 09 20:40:05.764924 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.764485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" event={"ID":"98d7898a-a2ae-45bf-9613-34429dc14823","Type":"ContainerDied","Data":"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677"} Sep 09 20:40:05.764924 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.764553 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc" event={"ID":"98d7898a-a2ae-45bf-9613-34429dc14823","Type":"ContainerDied","Data":"b1d4d1cec2b2264e0a23a27d38402f8a481626695563cbafad067918177ba67f"} Sep 09 20:40:05.764924 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.764575 2573 scope.go:117] "RemoveContainer" containerID="eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" Sep 09 20:40:05.774450 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.774297 2573 scope.go:117] "RemoveContainer" containerID="eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" Sep 09 20:40:05.774611 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:40:05.774591 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677\": container with ID starting with eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677 not found: ID does not exist" containerID="eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677" Sep 09 20:40:05.774668 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.774619 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677"} err="failed to get container status \"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677\": rpc error: code = NotFound desc = could not find container \"eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677\": container with ID starting with eb28085160975dd4f0b11ffc04d492f4706eb9e75d9488a84a21f42a3af1d677 not found: ID does not exist" Sep 09 20:40:05.784695 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.784667 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:40:05.788176 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:05.788151 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-49ftc"] Sep 09 20:40:06.015037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:06.014957 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" path="/var/lib/kubelet/pods/98d7898a-a2ae-45bf-9613-34429dc14823/volumes" Sep 09 20:40:24.719194 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.719161 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:40:24.719599 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.719548 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" Sep 09 20:40:24.719599 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.719561 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" Sep 09 20:40:24.719676 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.719628 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d7898a-a2ae-45bf-9613-34429dc14823" containerName="main" Sep 09 20:40:24.722866 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.722845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.725410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.725375 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\"" Sep 09 20:40:24.725572 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.725513 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\"" Sep 09 20:40:24.725782 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.725729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\"" Sep 09 20:40:24.725928 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.725859 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-znvvn\"" Sep 09 20:40:24.726130 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.726110 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:40:24.726293 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.726277 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\"" Sep 09 20:40:24.735076 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.735046 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:40:24.868150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fk4\" (UniqueName: \"kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.868586 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.868418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969338 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fk4\" (UniqueName: \"kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969654 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969881 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969736 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.969929 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.970011 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.969981 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.970291 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.970262 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.970424 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.970404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.971883 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.971856 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.972352 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.972336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:24.980494 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:24.980465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fk4\" (UniqueName: \"kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:25.034918 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:25.034874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:25.315592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:25.315564 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:40:25.318370 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:40:25.318339 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63a72e7_be8d_43f7_a988_f77934185aab.slice/crio-dd69d52d08c128e3ea6a7b7ec842f942d7b067d127df90d71513653bea2615dd WatchSource:0}: Error finding container dd69d52d08c128e3ea6a7b7ec842f942d7b067d127df90d71513653bea2615dd: Status 404 returned error can't find the container with id dd69d52d08c128e3ea6a7b7ec842f942d7b067d127df90d71513653bea2615dd Sep 09 20:40:25.867345 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:25.867304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerStarted","Data":"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210"} Sep 09 20:40:25.867816 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:25.867374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerStarted","Data":"dd69d52d08c128e3ea6a7b7ec842f942d7b067d127df90d71513653bea2615dd"} Sep 09 20:40:27.878820 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:27.878790 2573 generic.go:356] "Generic (PLEG): container finished" podID="f63a72e7-be8d-43f7-a988-f77934185aab" containerID="113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210" exitCode=0 Sep 09 20:40:27.879205 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:27.878862 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerDied","Data":"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210"} Sep 09 20:40:28.887244 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:28.887180 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerStarted","Data":"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3"} Sep 09 20:40:28.911568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:28.911505 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podStartSLOduration=2.581691711 podStartE2EDuration="4.911488565s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.329796854s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:28.909102108 +0000 UTC m=+2715.427981820" watchObservedRunningTime="2026-09-09 20:40:28.911488565 +0000 UTC m=+2715.430368318" Sep 09 20:40:29.035172 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:29.035130 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:35.036018 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:35.035974 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:40:35.037664 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:35.037630 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:40:45.036212 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:45.036170 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:40:55.035657 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:40:55.035609 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:03.395897 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.395860 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:41:03.398814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.398790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.402450 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.402426 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\"" Sep 09 20:41:03.402841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.402430 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\"" Sep 09 20:41:03.402841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.402445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-9wq6g\"" Sep 09 20:41:03.402841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.402544 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\"" Sep 09 20:41:03.402841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.402570 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-4f9dl\"" Sep 09 20:41:03.414465 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.414430 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:41:03.531939 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.531897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfq22\" (UniqueName: \"kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.532142 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.531981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.632912 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.632870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.633128 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.632974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfq22\" (UniqueName: \"kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.635609 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.635579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.641915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.641883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfq22\" (UniqueName: \"kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:03.722903 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:03.722798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:04.008348 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:04.008321 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:41:04.011008 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:41:04.010980 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a090966_b376_4bd8_8c6c_7d2888703b9d.slice/crio-a4d608fc7dd24e01a834c0a7e04ba6ec77ae97fdf35fb4559f437f3cd0f13682 WatchSource:0}: Error finding container a4d608fc7dd24e01a834c0a7e04ba6ec77ae97fdf35fb4559f437f3cd0f13682: Status 404 returned error can't find the container with id a4d608fc7dd24e01a834c0a7e04ba6ec77ae97fdf35fb4559f437f3cd0f13682 Sep 09 20:41:04.071892 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:04.071847 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" event={"ID":"9a090966-b376-4bd8-8c6c-7d2888703b9d","Type":"ContainerStarted","Data":"a4d608fc7dd24e01a834c0a7e04ba6ec77ae97fdf35fb4559f437f3cd0f13682"} Sep 09 20:41:05.035944 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:05.035903 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:05.079683 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:05.079650 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" event={"ID":"9a090966-b376-4bd8-8c6c-7d2888703b9d","Type":"ContainerStarted","Data":"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe"} Sep 09 20:41:05.079864 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:05.079755 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:05.100787 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:05.100730 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podStartSLOduration=2.100713327 podStartE2EDuration="2.100713327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:05.098414695 +0000 UTC m=+2751.617294399" watchObservedRunningTime="2026-09-09 20:41:05.100713327 +0000 UTC m=+2751.619593040" Sep 09 20:41:15.035730 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:15.035685 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:25.036196 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:25.036151 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:35.036395 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:35.036302 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:36.087440 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:36.087409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:41:45.036173 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:45.036128 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:41:55.035476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:41:55.035430 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:42:05.036028 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:05.035982 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:42:15.036287 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:15.036199 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:42:25.035759 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:25.035715 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:42:35.045279 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:35.045245 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:42:35.053148 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:35.053121 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:42:40.386734 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:40.386699 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:42:55.411830 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:55.411769 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" containerID="cri-o://855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3" gracePeriod=45 Sep 09 20:42:56.045983 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.045940 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused" Sep 09 20:42:56.166729 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.166703 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:42:56.288545 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288724 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288724 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288596 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288724 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288652 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288907 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288789 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288907 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288841 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9fk4\" (UniqueName: \"kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.288907 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288876 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.289067 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288923 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache" (OuterVolumeSpecName: "model-cache") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:42:56.289067 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.288964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location\") pod \"f63a72e7-be8d-43f7-a988-f77934185aab\" (UID: \"f63a72e7-be8d-43f7-a988-f77934185aab\") " Sep 09 20:42:56.289067 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.289018 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:42:56.289294 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.289273 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f63a72e7-be8d-43f7-a988-f77934185aab-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.289371 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.289299 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.289778 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.289748 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home" (OuterVolumeSpecName: "home") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:42:56.291310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.291278 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:42:56.291422 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.291395 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm" (OuterVolumeSpecName: "dshm") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:42:56.291475 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.291424 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4" (OuterVolumeSpecName: "kube-api-access-r9fk4") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "kube-api-access-r9fk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:42:56.302160 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.302124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:42:56.314947 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.314900 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f63a72e7-be8d-43f7-a988-f77934185aab" (UID: "f63a72e7-be8d-43f7-a988-f77934185aab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:42:56.390339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390303 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.390339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390334 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9fk4\" (UniqueName: \"kubernetes.io/projected/f63a72e7-be8d-43f7-a988-f77934185aab-kube-api-access-r9fk4\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.390339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390345 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.390580 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390357 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.390580 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390367 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a72e7-be8d-43f7-a988-f77934185aab-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.390580 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.390375 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f63a72e7-be8d-43f7-a988-f77934185aab-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:42:56.673485 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.673385 2573 generic.go:356] "Generic (PLEG): container finished" podID="f63a72e7-be8d-43f7-a988-f77934185aab" containerID="855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3" exitCode=0 Sep 09 20:42:56.673485 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.673447 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" Sep 09 20:42:56.673485 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.673462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerDied","Data":"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3"} Sep 09 20:42:56.674047 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.673508 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw" event={"ID":"f63a72e7-be8d-43f7-a988-f77934185aab","Type":"ContainerDied","Data":"dd69d52d08c128e3ea6a7b7ec842f942d7b067d127df90d71513653bea2615dd"} Sep 09 20:42:56.674047 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.673531 2573 scope.go:117] "RemoveContainer" containerID="855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3" Sep 09 20:42:56.683115 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.683096 2573 scope.go:117] "RemoveContainer" containerID="113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210" Sep 09 20:42:56.691833 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.691812 2573 scope.go:117] "RemoveContainer" containerID="855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3" Sep 09 20:42:56.692146 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:42:56.692128 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3\": container with ID starting with 855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3 not found: ID does not exist" containerID="855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3" Sep 09 20:42:56.692197 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.692156 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3"} err="failed to get container status \"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3\": rpc error: code = NotFound desc = could not find container \"855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3\": container with ID starting with 855de5e125036111eeccf775968857f96f1a58030d0af26ef383482e95f0e1b3 not found: ID does not exist" Sep 09 20:42:56.692197 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.692176 2573 scope.go:117] "RemoveContainer" containerID="113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210" Sep 09 20:42:56.692430 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:42:56.692414 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210\": container with ID starting with 113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210 not found: ID does not exist" containerID="113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210" Sep 09 20:42:56.692476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.692434 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210"} err="failed to get container status \"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210\": rpc error: code = NotFound desc = could not find container \"113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210\": container with ID starting with 113b91b8dc7d78b67244b24644e0383cb572cb17664769b2ded19d6a07ab0210 not found: ID does not exist" Sep 09 20:42:56.695591 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.695567 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:42:56.698600 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:56.698578 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b568fb4f887qw"] Sep 09 20:42:58.020699 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:42:58.020663 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" path="/var/lib/kubelet/pods/f63a72e7-be8d-43f7-a988-f77934185aab/volumes" Sep 09 20:43:07.384251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:07.384149 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:43:16.087075 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:16.087038 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:43:22.404985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:22.404913 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" containerID="cri-o://81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe" gracePeriod=45 Sep 09 20:43:26.087208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:26.087167 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:43:36.087083 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:36.087043 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:43:36.087532 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:36.087164 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:43:46.086783 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:46.086742 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:43:52.662284 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.662252 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:43:52.809461 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.809424 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs\") pod \"9a090966-b376-4bd8-8c6c-7d2888703b9d\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " Sep 09 20:43:52.809626 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.809585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfq22\" (UniqueName: \"kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22\") pod \"9a090966-b376-4bd8-8c6c-7d2888703b9d\" (UID: \"9a090966-b376-4bd8-8c6c-7d2888703b9d\") " Sep 09 20:43:52.811902 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.811871 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22" (OuterVolumeSpecName: "kube-api-access-dfq22") pod "9a090966-b376-4bd8-8c6c-7d2888703b9d" (UID: "9a090966-b376-4bd8-8c6c-7d2888703b9d"). InnerVolumeSpecName "kube-api-access-dfq22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:43:52.812026 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.811917 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a090966-b376-4bd8-8c6c-7d2888703b9d" (UID: "9a090966-b376-4bd8-8c6c-7d2888703b9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:43:52.911282 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.911217 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfq22\" (UniqueName: \"kubernetes.io/projected/9a090966-b376-4bd8-8c6c-7d2888703b9d-kube-api-access-dfq22\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:43:52.911282 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.911276 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a090966-b376-4bd8-8c6c-7d2888703b9d-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:43:52.948510 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.948474 2573 generic.go:356] "Generic (PLEG): container finished" podID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerID="81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe" exitCode=0 Sep 09 20:43:52.948692 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.948517 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" event={"ID":"9a090966-b376-4bd8-8c6c-7d2888703b9d","Type":"ContainerDied","Data":"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe"} Sep 09 20:43:52.948692 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.948560 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" event={"ID":"9a090966-b376-4bd8-8c6c-7d2888703b9d","Type":"ContainerDied","Data":"a4d608fc7dd24e01a834c0a7e04ba6ec77ae97fdf35fb4559f437f3cd0f13682"} Sep 09 20:43:52.948692 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.948581 2573 scope.go:117] "RemoveContainer" containerID="81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe" Sep 09 20:43:52.948692 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.948521 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9" Sep 09 20:43:52.958019 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.957999 2573 scope.go:117] "RemoveContainer" containerID="81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe" Sep 09 20:43:52.958366 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:43:52.958345 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe\": container with ID starting with 81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe not found: ID does not exist" containerID="81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe" Sep 09 20:43:52.958447 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.958377 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe"} err="failed to get container status \"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe\": rpc error: code = NotFound desc = could not find container \"81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe\": container with ID starting with 81df26e08bbe11b511378b30e5871d97d3d037fc0c161893c3a0a0c10bc058fe not found: ID does not exist" Sep 09 20:43:52.969971 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.969938 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:43:52.975100 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:52.975074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche87zd9"] Sep 09 20:43:54.015346 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:43:54.015310 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" path="/var/lib/kubelet/pods/9a090966-b376-4bd8-8c6c-7d2888703b9d/volumes" Sep 09 20:44:22.152210 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152174 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152578 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="storage-initializer" Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152591 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="storage-initializer" Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152603 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152610 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152629 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" Sep 09 20:44:22.152682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152634 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" Sep 09 20:44:22.152875 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152699 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63a72e7-be8d-43f7-a988-f77934185aab" containerName="main" Sep 09 20:44:22.152875 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.152712 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a090966-b376-4bd8-8c6c-7d2888703b9d" containerName="main" Sep 09 20:44:22.160375 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.160345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.162968 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.162911 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:44:22.163146 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.162984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\"" Sep 09 20:44:22.163146 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.162984 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\"" Sep 09 20:44:22.163146 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.163064 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\"" Sep 09 20:44:22.163395 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.163257 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\"" Sep 09 20:44:22.163468 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.163456 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-6mcxv\"" Sep 09 20:44:22.171094 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.171070 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:44:22.173039 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173164 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwth\" (UniqueName: \"kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173164 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173393 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.173393 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.173325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274279 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwth\" (UniqueName: \"kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274501 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274845 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274845 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274976 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.274976 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.274952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.275084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.275003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.275283 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.275262 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.276849 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.276829 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.277100 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.277082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.282064 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.282045 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwth\" (UniqueName: \"kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.473644 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.473543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:22.753628 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:22.753601 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:44:22.755775 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:44:22.755748 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ebf3c8_2a3c_4390_b935_1632048256b3.slice/crio-afefc03f858fa490ea675ebc3463dc6e9369a21c7d2c1534f96b592f5965c767 WatchSource:0}: Error finding container afefc03f858fa490ea675ebc3463dc6e9369a21c7d2c1534f96b592f5965c767: Status 404 returned error can't find the container with id afefc03f858fa490ea675ebc3463dc6e9369a21c7d2c1534f96b592f5965c767 Sep 09 20:44:23.094515 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:23.094480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerStarted","Data":"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc"} Sep 09 20:44:23.094715 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:23.094538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerStarted","Data":"afefc03f858fa490ea675ebc3463dc6e9369a21c7d2c1534f96b592f5965c767"} Sep 09 20:44:26.111119 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:26.111090 2573 generic.go:356] "Generic (PLEG): container finished" podID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerID="0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc" exitCode=0 Sep 09 20:44:26.111536 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:26.111136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerDied","Data":"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc"} Sep 09 20:44:27.118702 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:27.118664 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerStarted","Data":"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40"} Sep 09 20:44:27.144337 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:27.144284 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podStartSLOduration=2.403984623 podStartE2EDuration="5.144264166s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.740279543s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:27.142387115 +0000 UTC m=+2953.661266828" watchObservedRunningTime="2026-09-09 20:44:27.144264166 +0000 UTC m=+2953.663143879" Sep 09 20:44:27.474107 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:27.474014 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:32.473755 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:32.473721 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:44:32.475259 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:32.475212 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:44:42.474275 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:42.474215 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:44:52.474547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:44:52.474506 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:02.474333 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:02.474284 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:12.474105 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:12.474063 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:22.473983 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:22.473941 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:32.474018 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:32.473967 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:42.474634 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:42.474589 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused" Sep 09 20:45:52.484528 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:52.484496 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:45:52.492333 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:45:52.492311 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:46:21.609189 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:21.609109 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:46:36.632884 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:36.632823 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" containerID="cri-o://63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40" gracePeriod=45 Sep 09 20:46:37.293682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.293658 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:46:37.333759 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333730 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.333930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333802 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.333930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333823 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.333930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333843 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.333930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333858 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.333930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333892 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czwth\" (UniqueName: \"kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.334208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333938 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.334208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.333969 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location\") pod \"95ebf3c8-2a3c-4390-b935-1632048256b3\" (UID: \"95ebf3c8-2a3c-4390-b935-1632048256b3\") " Sep 09 20:46:37.334208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.334039 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:46:37.334418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.334302 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.334418 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.334361 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home" (OuterVolumeSpecName: "home") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:46:37.334860 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.334808 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:46:37.336699 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.336662 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth" (OuterVolumeSpecName: "kube-api-access-czwth") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "kube-api-access-czwth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:46:37.336950 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.336925 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:46:37.337062 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.336944 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm" (OuterVolumeSpecName: "dshm") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:46:37.352834 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.352793 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:46:37.367569 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.367497 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95ebf3c8-2a3c-4390-b935-1632048256b3" (UID: "95ebf3c8-2a3c-4390-b935-1632048256b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:46:37.435124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435086 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ebf3c8-2a3c-4390-b935-1632048256b3-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435118 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435126 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435133 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435433 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435143 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czwth\" (UniqueName: \"kubernetes.io/projected/95ebf3c8-2a3c-4390-b935-1632048256b3-kube-api-access-czwth\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435433 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435155 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95ebf3c8-2a3c-4390-b935-1632048256b3-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.435433 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.435162 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95ebf3c8-2a3c-4390-b935-1632048256b3-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:46:37.818313 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.818275 2573 generic.go:356] "Generic (PLEG): container finished" podID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerID="63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40" exitCode=0 Sep 09 20:46:37.818718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.818339 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" Sep 09 20:46:37.818718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.818360 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerDied","Data":"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40"} Sep 09 20:46:37.818718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.818440 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9" event={"ID":"95ebf3c8-2a3c-4390-b935-1632048256b3","Type":"ContainerDied","Data":"afefc03f858fa490ea675ebc3463dc6e9369a21c7d2c1534f96b592f5965c767"} Sep 09 20:46:37.818718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.818466 2573 scope.go:117] "RemoveContainer" containerID="63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40" Sep 09 20:46:37.828411 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.828391 2573 scope.go:117] "RemoveContainer" containerID="0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc" Sep 09 20:46:37.837585 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.837566 2573 scope.go:117] "RemoveContainer" containerID="63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40" Sep 09 20:46:37.837878 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:46:37.837854 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40\": container with ID starting with 63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40 not found: ID does not exist" containerID="63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40" Sep 09 20:46:37.837930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.837889 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40"} err="failed to get container status \"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40\": rpc error: code = NotFound desc = could not find container \"63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40\": container with ID starting with 63f236d48acb421b465ccbf43ae5a0a81d8ff0db3114d1c4a567d0a90e846d40 not found: ID does not exist" Sep 09 20:46:37.837930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.837907 2573 scope.go:117] "RemoveContainer" containerID="0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc" Sep 09 20:46:37.838149 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:46:37.838130 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc\": container with ID starting with 0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc not found: ID does not exist" containerID="0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc" Sep 09 20:46:37.838200 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.838156 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc"} err="failed to get container status \"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc\": rpc error: code = NotFound desc = could not find container \"0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc\": container with ID starting with 0ed30f6285725d74a92777c8b4dbb4b95e521cde3e6b13faaafe82289ff94acc not found: ID does not exist" Sep 09 20:46:37.858555 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.858520 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:46:37.872631 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:37.872600 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dbff548f7-xzph9"] Sep 09 20:46:38.015549 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:46:38.015508 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" path="/var/lib/kubelet/pods/95ebf3c8-2a3c-4390-b935-1632048256b3/volumes" Sep 09 20:47:15.047360 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047326 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:47:15.047773 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047691 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" Sep 09 20:47:15.047773 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047702 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" Sep 09 20:47:15.047773 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047714 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="storage-initializer" Sep 09 20:47:15.047773 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047720 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="storage-initializer" Sep 09 20:47:15.047904 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.047794 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95ebf3c8-2a3c-4390-b935-1632048256b3" containerName="main" Sep 09 20:47:15.052042 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.052023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.061725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.061702 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:47:15.062133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.062119 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Sep 09 20:47:15.062193 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.062172 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\"" Sep 09 20:47:15.070156 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.070136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\"" Sep 09 20:47:15.071141 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.071125 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-4k47c\"" Sep 09 20:47:15.071568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.071552 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\"" Sep 09 20:47:15.077088 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvj2v\" (UniqueName: \"kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077148 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077148 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077252 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077252 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077353 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077353 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.077353 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.077314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.107151 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.107121 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:47:15.178186 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178186 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvj2v\" (UniqueName: \"kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178794 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178794 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178794 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.178948 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.178883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.179025 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.179006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.180858 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.180835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.181091 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.181070 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.214651 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.214614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvj2v\" (UniqueName: \"kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v\") pod \"custom-route-timeout-test-kserve-767dfc69d4-khkkk\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.362287 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.362197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:15.680904 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:15.680818 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:47:15.684210 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:47:15.684184 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b73e3c_1172_4250_be9a_4ac1327ede63.slice/crio-fe9979058f9ef620034abf948a58e506703d8088db49ac8fcb779fc6df17dcf1 WatchSource:0}: Error finding container fe9979058f9ef620034abf948a58e506703d8088db49ac8fcb779fc6df17dcf1: Status 404 returned error can't find the container with id fe9979058f9ef620034abf948a58e506703d8088db49ac8fcb779fc6df17dcf1 Sep 09 20:47:16.004346 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:16.004247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerStarted","Data":"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f"} Sep 09 20:47:16.004346 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:16.004300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerStarted","Data":"fe9979058f9ef620034abf948a58e506703d8088db49ac8fcb779fc6df17dcf1"} Sep 09 20:47:19.024044 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:19.024009 2573 generic.go:356] "Generic (PLEG): container finished" podID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerID="967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f" exitCode=0 Sep 09 20:47:19.024466 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:19.024084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerDied","Data":"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f"} Sep 09 20:47:20.030522 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:20.030486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerStarted","Data":"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1"} Sep 09 20:47:20.083033 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:20.082968 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podStartSLOduration=3.8008572259999998 podStartE2EDuration="6.082952483s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282095257s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:20.08189842 +0000 UTC m=+3126.600778163" watchObservedRunningTime="2026-09-09 20:47:20.082952483 +0000 UTC m=+3126.601832195" Sep 09 20:47:20.362725 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:20.362680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:25.363117 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:25.363074 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:47:25.364865 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:25.364835 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:47:35.363638 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:35.363582 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:47:45.363211 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:45.363156 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:47:55.363483 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:47:55.363438 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:05.362908 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:05.362866 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:15.363300 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:15.363244 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:25.363506 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:25.363459 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:35.362926 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:35.362864 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:45.363568 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:45.363509 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:48:55.363289 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:48:55.363246 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:49:05.363565 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:05.363451 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:49:15.363545 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:15.363503 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:49:25.373606 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:25.373561 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:49:25.381966 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:25.381932 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:49:30.826560 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:30.826522 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:49:45.850316 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:45.850266 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" containerID="cri-o://07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1" gracePeriod=45 Sep 09 20:49:46.374525 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.374481 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused" Sep 09 20:49:46.604134 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.604101 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:49:46.608921 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.608899 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.608927 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.608958 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.608983 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609012 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609069 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609058 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609364 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609090 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609364 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609147 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvj2v\" (UniqueName: \"kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v\") pod \"95b73e3c-1172-4250-be9a-4ac1327ede63\" (UID: \"95b73e3c-1172-4250-be9a-4ac1327ede63\") " Sep 09 20:49:46.609364 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609244 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache" (OuterVolumeSpecName: "model-cache") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:49:46.609562 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609435 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.609785 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609756 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:49:46.610006 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.609982 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home" (OuterVolumeSpecName: "home") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:49:46.611763 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.611737 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:49:46.611958 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.611937 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm" (OuterVolumeSpecName: "dshm") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:49:46.611958 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.611946 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v" (OuterVolumeSpecName: "kube-api-access-hvj2v") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "kube-api-access-hvj2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:49:46.621951 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.621925 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:49:46.637458 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.637373 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95b73e3c-1172-4250-be9a-4ac1327ede63" (UID: "95b73e3c-1172-4250-be9a-4ac1327ede63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:49:46.710082 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710040 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710082 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710074 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710082 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710085 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95b73e3c-1172-4250-be9a-4ac1327ede63-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710082 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710095 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b73e3c-1172-4250-be9a-4ac1327ede63-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710406 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710102 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710406 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710110 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvj2v\" (UniqueName: \"kubernetes.io/projected/95b73e3c-1172-4250-be9a-4ac1327ede63-kube-api-access-hvj2v\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.710406 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.710118 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95b73e3c-1172-4250-be9a-4ac1327ede63-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:49:46.803149 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.803109 2573 generic.go:356] "Generic (PLEG): container finished" podID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerID="07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1" exitCode=0 Sep 09 20:49:46.803358 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.803186 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" Sep 09 20:49:46.803358 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.803189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerDied","Data":"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1"} Sep 09 20:49:46.803358 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.803276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk" event={"ID":"95b73e3c-1172-4250-be9a-4ac1327ede63","Type":"ContainerDied","Data":"fe9979058f9ef620034abf948a58e506703d8088db49ac8fcb779fc6df17dcf1"} Sep 09 20:49:46.803358 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.803299 2573 scope.go:117] "RemoveContainer" containerID="07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1" Sep 09 20:49:46.812759 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.812740 2573 scope.go:117] "RemoveContainer" containerID="967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f" Sep 09 20:49:46.820178 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.820161 2573 scope.go:117] "RemoveContainer" containerID="07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1" Sep 09 20:49:46.820464 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:49:46.820447 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1\": container with ID starting with 07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1 not found: ID does not exist" containerID="07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1" Sep 09 20:49:46.820539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.820471 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1"} err="failed to get container status \"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1\": rpc error: code = NotFound desc = could not find container \"07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1\": container with ID starting with 07c45a188a3ee9f2ee66483567ac848e47c44ffb51de6c35b175347f57b615d1 not found: ID does not exist" Sep 09 20:49:46.820539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.820487 2573 scope.go:117] "RemoveContainer" containerID="967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f" Sep 09 20:49:46.820708 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:49:46.820694 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f\": container with ID starting with 967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f not found: ID does not exist" containerID="967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f" Sep 09 20:49:46.820748 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.820711 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f"} err="failed to get container status \"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f\": rpc error: code = NotFound desc = could not find container \"967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f\": container with ID starting with 967857faac62d7073f9a65f7d771de5db6f03812d439732b9ec2d4631d4ae61f not found: ID does not exist" Sep 09 20:49:46.826170 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.826149 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:49:46.830747 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:46.830727 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-767dfc69d4-khkkk"] Sep 09 20:49:48.015575 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:49:48.015528 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" path="/var/lib/kubelet/pods/95b73e3c-1172-4250-be9a-4ac1327ede63/volumes" Sep 09 20:50:07.878341 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878302 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:07.878804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878701 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="storage-initializer" Sep 09 20:50:07.878804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878715 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="storage-initializer" Sep 09 20:50:07.878804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878734 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" Sep 09 20:50:07.878804 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878740 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" Sep 09 20:50:07.878950 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.878816 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b73e3c-1172-4250-be9a-4ac1327ede63" containerName="main" Sep 09 20:50:07.881770 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.881750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:07.886624 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.886596 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-lc4g5\"" Sep 09 20:50:07.886789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.886598 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-xtqmb\"" Sep 09 20:50:07.886789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.886620 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\"" Sep 09 20:50:07.886789 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.886652 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\"" Sep 09 20:50:07.886985 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.886654 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\"" Sep 09 20:50:07.893922 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:07.893891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:08.006828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.006785 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88qk\" (UniqueName: \"kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.006828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.006834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.108184 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.108140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88qk\" (UniqueName: \"kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.108184 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.108193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.110917 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.110893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.116006 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.115984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88qk\" (UniqueName: \"kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.193668 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.193561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.471433 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.471347 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:08.474747 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:50:08.474721 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183a455_5e7b_4832_a760_376551d9ce4b.slice/crio-81eecd8dc225a6868bb1607d8c3c62b9b8ed65e847d383b27bc3e790f1f80aff WatchSource:0}: Error finding container 81eecd8dc225a6868bb1607d8c3c62b9b8ed65e847d383b27bc3e790f1f80aff: Status 404 returned error can't find the container with id 81eecd8dc225a6868bb1607d8c3c62b9b8ed65e847d383b27bc3e790f1f80aff Sep 09 20:50:08.923037 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.923000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" event={"ID":"2183a455-5e7b-4832-a760-376551d9ce4b","Type":"ContainerStarted","Data":"6c043aaf03689e9a804485243f9e7da8dd7b634fa66285be6e1ddf5c80c0450f"} Sep 09 20:50:08.923641 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.923052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" event={"ID":"2183a455-5e7b-4832-a760-376551d9ce4b","Type":"ContainerStarted","Data":"81eecd8dc225a6868bb1607d8c3c62b9b8ed65e847d383b27bc3e790f1f80aff"} Sep 09 20:50:08.923641 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.923117 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:08.942904 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:08.942841 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" podStartSLOduration=1.9428255760000002 podStartE2EDuration="1.942825576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:08.940923106 +0000 UTC m=+3295.459802820" watchObservedRunningTime="2026-09-09 20:50:08.942825576 +0000 UTC m=+3295.461705288" Sep 09 20:50:09.860584 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:09.860550 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:09.928347 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:09.928313 2573 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" secret="" err="secret \"e2e-del-blocked-svc-epp-sa-dockercfg-lc4g5\" not found" Sep 09 20:50:10.026862 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:10.026823 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:10.027017 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:10.026913 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:10.526890276 +0000 UTC m=+3297.045770178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:10.531569 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:10.531527 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:10.531752 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:10.531610 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:11.531589825 +0000 UTC m=+3298.050469516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:11.542876 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:11.542838 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:11.543299 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:11.542913 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:13.54289907 +0000 UTC m=+3300.061778761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:13.561384 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:13.561346 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:13.561793 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:13.561420 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:17.5614054 +0000 UTC m=+3304.080285090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:17.601119 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:17.601081 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:17.601553 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:17.601159 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:25.601140766 +0000 UTC m=+3312.120020478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:25.678289 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:25.678256 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:25.678693 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:25.678350 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:41.678335251 +0000 UTC m=+3328.197214942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:25.953216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:25.953122 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" containerID="cri-o://6c043aaf03689e9a804485243f9e7da8dd7b634fa66285be6e1ddf5c80c0450f" gracePeriod=45 Sep 09 20:50:40.935601 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:40.935559 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:50:41.729138 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:41.729074 2573 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:41.729383 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:50:41.729193 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs podName:2183a455-5e7b-4832-a760-376551d9ce4b nodeName:}" failed. No retries permitted until 2026-09-09 20:51:13.729173295 +0000 UTC m=+3360.248052986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs") pod "e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found Sep 09 20:50:49.246990 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.246951 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:50:49.250590 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.250568 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.253404 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.253375 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\"" Sep 09 20:50:49.253682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.253658 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\"" Sep 09 20:50:49.253823 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.253692 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:50:49.253917 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.253895 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-sjljx\"" Sep 09 20:50:49.253994 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.253923 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\"" Sep 09 20:50:49.254457 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.254437 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\"" Sep 09 20:50:49.267126 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.267098 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:50:49.398946 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.398906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.398946 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.398942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399169 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.398967 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399169 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.399093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399169 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.399125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399169 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.399148 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75pb\" (UniqueName: \"kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399349 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.399213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.399349 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.399293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500271 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500517 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75pb\" (UniqueName: \"kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500923 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500701 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500923 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500923 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500852 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.500923 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.500910 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.501098 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.501069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.502961 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.502934 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.503187 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.503166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.517841 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.517807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75pb\" (UniqueName: \"kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb\") pod \"router-with-refs-test-kserve-f765b7c5b-46vpx\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.561302 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.561262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:49.850646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:49.850618 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:50:49.853708 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:50:49.853684 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2516cc6d_9c24_48d6_b357_ddc6a23cba16.slice/crio-82824f8adb2472e18c83fc2b29ef7680af03b15d1f14600e961cfe1caf8dfdd4 WatchSource:0}: Error finding container 82824f8adb2472e18c83fc2b29ef7680af03b15d1f14600e961cfe1caf8dfdd4: Status 404 returned error can't find the container with id 82824f8adb2472e18c83fc2b29ef7680af03b15d1f14600e961cfe1caf8dfdd4 Sep 09 20:50:50.131408 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:50.131306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerStarted","Data":"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11"} Sep 09 20:50:50.131408 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:50.131365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerStarted","Data":"82824f8adb2472e18c83fc2b29ef7680af03b15d1f14600e961cfe1caf8dfdd4"} Sep 09 20:50:50.934790 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:50.934748 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:50:53.149208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:53.149175 2573 generic.go:356] "Generic (PLEG): container finished" podID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerID="76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11" exitCode=0 Sep 09 20:50:53.149208 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:53.149215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerDied","Data":"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11"} Sep 09 20:50:54.156422 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:54.156387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerStarted","Data":"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8"} Sep 09 20:50:54.184814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:54.184752 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podStartSLOduration=2.960048655 podStartE2EDuration="5.184734756s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.224686101s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:54.182859606 +0000 UTC m=+3340.701739319" watchObservedRunningTime="2026-09-09 20:50:54.184734756 +0000 UTC m=+3340.703614468" Sep 09 20:50:54.561980 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:54.561946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:56.169472 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.169435 2573 generic.go:356] "Generic (PLEG): container finished" podID="2183a455-5e7b-4832-a760-376551d9ce4b" containerID="6c043aaf03689e9a804485243f9e7da8dd7b634fa66285be6e1ddf5c80c0450f" exitCode=0 Sep 09 20:50:56.169838 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.169509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" event={"ID":"2183a455-5e7b-4832-a760-376551d9ce4b","Type":"ContainerDied","Data":"6c043aaf03689e9a804485243f9e7da8dd7b634fa66285be6e1ddf5c80c0450f"} Sep 09 20:50:56.202567 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.202537 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:56.384454 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.384419 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs\") pod \"2183a455-5e7b-4832-a760-376551d9ce4b\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " Sep 09 20:50:56.384653 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.384554 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p88qk\" (UniqueName: \"kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk\") pod \"2183a455-5e7b-4832-a760-376551d9ce4b\" (UID: \"2183a455-5e7b-4832-a760-376551d9ce4b\") " Sep 09 20:50:56.386869 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.386836 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk" (OuterVolumeSpecName: "kube-api-access-p88qk") pod "2183a455-5e7b-4832-a760-376551d9ce4b" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b"). InnerVolumeSpecName "kube-api-access-p88qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:50:56.387005 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.386916 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2183a455-5e7b-4832-a760-376551d9ce4b" (UID: "2183a455-5e7b-4832-a760-376551d9ce4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:50:56.486175 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.486132 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88qk\" (UniqueName: \"kubernetes.io/projected/2183a455-5e7b-4832-a760-376551d9ce4b-kube-api-access-p88qk\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:50:56.486175 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:56.486166 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2183a455-5e7b-4832-a760-376551d9ce4b-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:50:57.176762 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:57.176730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" event={"ID":"2183a455-5e7b-4832-a760-376551d9ce4b","Type":"ContainerDied","Data":"81eecd8dc225a6868bb1607d8c3c62b9b8ed65e847d383b27bc3e790f1f80aff"} Sep 09 20:50:57.177251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:57.176781 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z" Sep 09 20:50:57.177251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:57.176800 2573 scope.go:117] "RemoveContainer" containerID="6c043aaf03689e9a804485243f9e7da8dd7b634fa66285be6e1ddf5c80c0450f" Sep 09 20:50:57.206975 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:57.206944 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:57.218150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:57.218119 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-tl92z"] Sep 09 20:50:58.015881 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:58.015844 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" path="/var/lib/kubelet/pods/2183a455-5e7b-4832-a760-376551d9ce4b/volumes" Sep 09 20:50:59.562253 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:59.562188 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:50:59.563718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:50:59.563687 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:09.561923 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:09.561874 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:16.164245 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.164182 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:51:16.164835 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.164812 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" Sep 09 20:51:16.164880 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.164842 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" Sep 09 20:51:16.164966 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.164952 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2183a455-5e7b-4832-a760-376551d9ce4b" containerName="main" Sep 09 20:51:16.169827 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.169802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.173429 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.173396 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\"" Sep 09 20:51:16.173664 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.173644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6htds\"" Sep 09 20:51:16.173751 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.173644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-jxdk2\"" Sep 09 20:51:16.173751 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.173696 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\"" Sep 09 20:51:16.173751 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.173694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\"" Sep 09 20:51:16.177710 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.177682 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:51:16.255159 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.255107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.255402 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.255187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf6l\" (UniqueName: \"kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.356448 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.356411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.356636 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.356454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf6l\" (UniqueName: \"kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.359350 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.359313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.365992 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.365963 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf6l\" (UniqueName: \"kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.482143 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.482053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:16.758969 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:16.758873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:51:16.763023 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:51:16.762989 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22a199d_edd7_48bb_8c89_058eea97c25c.slice/crio-65d50d4d080f1b30c272379e9822ae6277d5744f9c4108f19a7dc1de2c316a5a WatchSource:0}: Error finding container 65d50d4d080f1b30c272379e9822ae6277d5744f9c4108f19a7dc1de2c316a5a: Status 404 returned error can't find the container with id 65d50d4d080f1b30c272379e9822ae6277d5744f9c4108f19a7dc1de2c316a5a Sep 09 20:51:17.282378 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:17.282343 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" event={"ID":"c22a199d-edd7-48bb-8c89-058eea97c25c","Type":"ContainerStarted","Data":"c3a5b9157842add3c6ab6e38932c5abe5ad6493ae320468d09402c9f791504a2"} Sep 09 20:51:17.282792 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:17.282399 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" event={"ID":"c22a199d-edd7-48bb-8c89-058eea97c25c","Type":"ContainerStarted","Data":"65d50d4d080f1b30c272379e9822ae6277d5744f9c4108f19a7dc1de2c316a5a"} Sep 09 20:51:17.282792 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:17.282427 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:17.307411 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:17.307343 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podStartSLOduration=1.307294657 podStartE2EDuration="1.307294657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:17.30670572 +0000 UTC m=+3363.825585443" watchObservedRunningTime="2026-09-09 20:51:17.307294657 +0000 UTC m=+3363.826174370" Sep 09 20:51:19.562466 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:19.562419 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:29.562405 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:29.562362 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:39.561861 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:39.561808 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:48.289740 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:48.289708 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:51:49.561646 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:49.561602 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:51:59.562150 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:51:59.562097 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:09.561661 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:09.561578 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:19.562656 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:19.562610 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:29.562383 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:29.562327 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:39.562721 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:39.562670 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:49.561672 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:49.561624 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused" Sep 09 20:52:59.572015 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:59.571978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:52:59.579953 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:52:59.579921 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:53:04.711084 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:04.711043 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:53:18.460564 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:18.460525 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:53:19.735214 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:19.735170 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" containerID="cri-o://253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8" gracePeriod=45 Sep 09 20:53:20.498332 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.498305 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:53:20.529348 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529316 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529546 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529546 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529398 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529546 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529424 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529546 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529478 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j75pb\" (UniqueName: \"kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529546 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529502 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529552 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529588 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert\") pod \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\" (UID: \"2516cc6d-9c24-48d6-b357-ddc6a23cba16\") " Sep 09 20:53:20.529828 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529729 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache" (OuterVolumeSpecName: "model-cache") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:53:20.530000 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.529952 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.530322 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.530058 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:53:20.530597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.530555 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home" (OuterVolumeSpecName: "home") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:53:20.532363 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.532324 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:53:20.532487 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.532403 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm" (OuterVolumeSpecName: "dshm") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:53:20.533319 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.533289 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb" (OuterVolumeSpecName: "kube-api-access-j75pb") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "kube-api-access-j75pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:53:20.543419 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.543379 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:53:20.560319 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.560267 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2516cc6d-9c24-48d6-b357-ddc6a23cba16" (UID: "2516cc6d-9c24-48d6-b357-ddc6a23cba16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631117 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631158 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631175 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631190 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75pb\" (UniqueName: \"kubernetes.io/projected/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kube-api-access-j75pb\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631205 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631218 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2516cc6d-9c24-48d6-b357-ddc6a23cba16-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.631579 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.631272 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2516cc6d-9c24-48d6-b357-ddc6a23cba16-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:53:20.929596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.929502 2573 generic.go:356] "Generic (PLEG): container finished" podID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerID="253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8" exitCode=0 Sep 09 20:53:20.929596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.929566 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" Sep 09 20:53:20.929596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.929575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerDied","Data":"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8"} Sep 09 20:53:20.930411 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.929618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx" event={"ID":"2516cc6d-9c24-48d6-b357-ddc6a23cba16","Type":"ContainerDied","Data":"82824f8adb2472e18c83fc2b29ef7680af03b15d1f14600e961cfe1caf8dfdd4"} Sep 09 20:53:20.930411 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.929639 2573 scope.go:117] "RemoveContainer" containerID="253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8" Sep 09 20:53:20.939476 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.939456 2573 scope.go:117] "RemoveContainer" containerID="76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11" Sep 09 20:53:20.948565 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.948543 2573 scope.go:117] "RemoveContainer" containerID="253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8" Sep 09 20:53:20.948911 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:53:20.948893 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8\": container with ID starting with 253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8 not found: ID does not exist" containerID="253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8" Sep 09 20:53:20.948997 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.948923 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8"} err="failed to get container status \"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8\": rpc error: code = NotFound desc = could not find container \"253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8\": container with ID starting with 253c07b58640e370ba952b6e571d999a45036f0d66529478f90b59e346d77cd8 not found: ID does not exist" Sep 09 20:53:20.948997 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.948951 2573 scope.go:117] "RemoveContainer" containerID="76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11" Sep 09 20:53:20.949239 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:53:20.949206 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11\": container with ID starting with 76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11 not found: ID does not exist" containerID="76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11" Sep 09 20:53:20.949288 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.949250 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11"} err="failed to get container status \"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11\": rpc error: code = NotFound desc = could not find container \"76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11\": container with ID starting with 76e8586dd1be1cef1bc4263e4a2084195fd3aa03ff65ce41951a820cee301e11 not found: ID does not exist" Sep 09 20:53:20.960245 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.960175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:53:20.965607 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:20.965577 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-f765b7c5b-46vpx"] Sep 09 20:53:22.015120 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:22.015086 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" path="/var/lib/kubelet/pods/2516cc6d-9c24-48d6-b357-ddc6a23cba16/volumes" Sep 09 20:53:28.289981 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:28.289934 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:53:33.938164 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:33.938119 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" containerID="cri-o://c3a5b9157842add3c6ab6e38932c5abe5ad6493ae320468d09402c9f791504a2" gracePeriod=45 Sep 09 20:53:38.289067 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:38.288984 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:53:48.289523 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:48.289483 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:53:48.289934 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:48.289598 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:53:58.289781 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:53:58.289740 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Sep 09 20:54:04.149690 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.149656 2573 generic.go:356] "Generic (PLEG): container finished" podID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerID="c3a5b9157842add3c6ab6e38932c5abe5ad6493ae320468d09402c9f791504a2" exitCode=0 Sep 09 20:54:04.150097 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.149731 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" event={"ID":"c22a199d-edd7-48bb-8c89-058eea97c25c","Type":"ContainerDied","Data":"c3a5b9157842add3c6ab6e38932c5abe5ad6493ae320468d09402c9f791504a2"} Sep 09 20:54:04.195441 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.195415 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:54:04.238767 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.238687 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jf6l\" (UniqueName: \"kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l\") pod \"c22a199d-edd7-48bb-8c89-058eea97c25c\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " Sep 09 20:54:04.238767 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.238731 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs\") pod \"c22a199d-edd7-48bb-8c89-058eea97c25c\" (UID: \"c22a199d-edd7-48bb-8c89-058eea97c25c\") " Sep 09 20:54:04.241087 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.241062 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c22a199d-edd7-48bb-8c89-058eea97c25c" (UID: "c22a199d-edd7-48bb-8c89-058eea97c25c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:54:04.241087 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.241069 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l" (OuterVolumeSpecName: "kube-api-access-6jf6l") pod "c22a199d-edd7-48bb-8c89-058eea97c25c" (UID: "c22a199d-edd7-48bb-8c89-058eea97c25c"). InnerVolumeSpecName "kube-api-access-6jf6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:54:04.340032 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.339998 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jf6l\" (UniqueName: \"kubernetes.io/projected/c22a199d-edd7-48bb-8c89-058eea97c25c-kube-api-access-6jf6l\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:54:04.340032 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:04.340029 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c22a199d-edd7-48bb-8c89-058eea97c25c-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:54:05.156197 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:05.156166 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" Sep 09 20:54:05.156610 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:05.156166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g" event={"ID":"c22a199d-edd7-48bb-8c89-058eea97c25c","Type":"ContainerDied","Data":"65d50d4d080f1b30c272379e9822ae6277d5744f9c4108f19a7dc1de2c316a5a"} Sep 09 20:54:05.156610 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:05.156302 2573 scope.go:117] "RemoveContainer" containerID="c3a5b9157842add3c6ab6e38932c5abe5ad6493ae320468d09402c9f791504a2" Sep 09 20:54:05.206935 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:05.206904 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:54:05.212401 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:05.212369 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f4g7g"] Sep 09 20:54:06.014810 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:06.014771 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" path="/var/lib/kubelet/pods/c22a199d-edd7-48bb-8c89-058eea97c25c/volumes" Sep 09 20:54:12.103579 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.103539 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:54:12.104137 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104119 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="storage-initializer" Sep 09 20:54:12.104200 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104141 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="storage-initializer" Sep 09 20:54:12.104200 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104193 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" Sep 09 20:54:12.104305 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104202 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" Sep 09 20:54:12.104305 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104217 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" Sep 09 20:54:12.104305 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104243 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" Sep 09 20:54:12.104402 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104357 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22a199d-edd7-48bb-8c89-058eea97c25c" containerName="main" Sep 09 20:54:12.104402 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.104372 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2516cc6d-9c24-48d6-b357-ddc6a23cba16" containerName="main" Sep 09 20:54:12.109752 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.109729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.115320 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.115291 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\"" Sep 09 20:54:12.115553 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.115316 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\"" Sep 09 20:54:12.115553 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.115323 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\"" Sep 09 20:54:12.115682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.115601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrrxw\"" Sep 09 20:54:12.115682 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.115631 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-xn895\"" Sep 09 20:54:12.117871 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.117854 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\"" Sep 09 20:54:12.119178 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.119161 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:54:12.128976 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.128939 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:54:12.213624 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213624 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213850 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213850 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213850 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4qj\" (UniqueName: \"kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.213974 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.213921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.314764 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.314764 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4qj\" (UniqueName: \"kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.314977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315078 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315469 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315589 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.315637 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.315596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.317381 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.317353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.317670 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.317651 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.324977 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.324950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4qj\" (UniqueName: \"kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.420321 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.420210 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:12.708767 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.708738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:54:12.710783 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:54:12.710756 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af88233_73b0_44df_a363_c00e1fe85faa.slice/crio-25161896fbb635455d1e9771810bc7d609f0c355e1c9749f3f8f1f4334354675 WatchSource:0}: Error finding container 25161896fbb635455d1e9771810bc7d609f0c355e1c9749f3f8f1f4334354675: Status 404 returned error can't find the container with id 25161896fbb635455d1e9771810bc7d609f0c355e1c9749f3f8f1f4334354675 Sep 09 20:54:12.712932 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:12.712911 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:54:13.198121 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:13.198084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerStarted","Data":"25161896fbb635455d1e9771810bc7d609f0c355e1c9749f3f8f1f4334354675"} Sep 09 20:54:15.209453 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:15.209417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerStarted","Data":"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a"} Sep 09 20:54:15.209835 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:15.209561 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:15.212935 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:15.212916 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\"" Sep 09 20:54:16.220105 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:16.220049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerStarted","Data":"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b"} Sep 09 20:54:18.234301 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:18.234263 2573 generic.go:356] "Generic (PLEG): container finished" podID="4af88233-73b0-44df-a363-c00e1fe85faa" containerID="15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b" exitCode=0 Sep 09 20:54:18.234701 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:18.234332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerDied","Data":"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b"} Sep 09 20:54:19.242596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:19.242560 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerStarted","Data":"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0"} Sep 09 20:54:19.277570 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:19.277507 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podStartSLOduration=2.791411524 podStartE2EDuration="7.277488973s" totalImagesPullingTime="2.280967997s" totalInitContainerRuntime="2.205109452s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:19.275845656 +0000 UTC m=+3545.794725368" watchObservedRunningTime="2026-09-09 20:54:19.277488973 +0000 UTC m=+3545.796368686" Sep 09 20:54:22.421071 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:22.421035 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:22.421490 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:22.421125 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:22.422539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:22.422515 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:54:32.420992 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:32.420949 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:54:33.281766 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:33.281734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:54:42.421433 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:42.421388 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:54:52.421561 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:54:52.421493 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:02.421580 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:02.421524 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:12.421486 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:12.421431 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:22.421644 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:22.421533 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:32.420946 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:32.420897 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:42.421096 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:42.421043 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:55:52.420822 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:55:52.420762 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:56:02.421254 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:02.421170 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:56:12.421596 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:12.421534 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused" Sep 09 20:56:22.436507 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:22.436471 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:56:22.448735 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:22.448704 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:56:32.430175 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:32.430133 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:56:47.456858 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:47.456818 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" containerID="cri-o://9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" gracePeriod=45 Sep 09 20:56:48.069419 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.069366 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="llm-d-routing-sidecar" containerID="cri-o://c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" gracePeriod=45 Sep 09 20:56:48.322354 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.322330 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:56:48.501354 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501318 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl4qj\" (UniqueName: \"kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501379 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501443 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501486 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501552 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501575 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home\") pod \"4af88233-73b0-44df-a363-c00e1fe85faa\" (UID: \"4af88233-73b0-44df-a363-c00e1fe85faa\") " Sep 09 20:56:48.501868 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.501839 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:48.502553 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.502387 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home" (OuterVolumeSpecName: "home") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:48.503774 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.503744 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:48.504016 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.503985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj" (OuterVolumeSpecName: "kube-api-access-dl4qj") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "kube-api-access-dl4qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:56:48.504273 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.504244 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:48.504273 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.504265 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm" (OuterVolumeSpecName: "dshm") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:48.514751 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.514719 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:48.527201 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.527172 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4af88233-73b0-44df-a363-c00e1fe85faa" (UID: "4af88233-73b0-44df-a363-c00e1fe85faa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602492 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602529 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af88233-73b0-44df-a363-c00e1fe85faa-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602542 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af88233-73b0-44df-a363-c00e1fe85faa-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602552 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602563 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602574 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602586 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4af88233-73b0-44df-a363-c00e1fe85faa-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:48.602597 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:48.602597 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4qj\" (UniqueName: \"kubernetes.io/projected/4af88233-73b0-44df-a363-c00e1fe85faa-kube-api-access-dl4qj\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:56:49.044813 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044781 2573 generic.go:356] "Generic (PLEG): container finished" podID="4af88233-73b0-44df-a363-c00e1fe85faa" containerID="9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" exitCode=0 Sep 09 20:56:49.044813 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044808 2573 generic.go:356] "Generic (PLEG): container finished" podID="4af88233-73b0-44df-a363-c00e1fe85faa" containerID="c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" exitCode=0 Sep 09 20:56:49.045031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044836 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" Sep 09 20:56:49.045031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerDied","Data":"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0"} Sep 09 20:56:49.045031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerDied","Data":"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a"} Sep 09 20:56:49.045031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws" event={"ID":"4af88233-73b0-44df-a363-c00e1fe85faa","Type":"ContainerDied","Data":"25161896fbb635455d1e9771810bc7d609f0c355e1c9749f3f8f1f4334354675"} Sep 09 20:56:49.045031 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.044928 2573 scope.go:117] "RemoveContainer" containerID="9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" Sep 09 20:56:49.054853 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.054831 2573 scope.go:117] "RemoveContainer" containerID="15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b" Sep 09 20:56:49.065600 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.065577 2573 scope.go:117] "RemoveContainer" containerID="c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" Sep 09 20:56:49.072014 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.071980 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:56:49.075124 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.075104 2573 scope.go:117] "RemoveContainer" containerID="9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" Sep 09 20:56:49.075470 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:56:49.075448 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0\": container with ID starting with 9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0 not found: ID does not exist" containerID="9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" Sep 09 20:56:49.075569 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.075478 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0"} err="failed to get container status \"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0\": rpc error: code = NotFound desc = could not find container \"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0\": container with ID starting with 9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0 not found: ID does not exist" Sep 09 20:56:49.075569 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.075500 2573 scope.go:117] "RemoveContainer" containerID="15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b" Sep 09 20:56:49.075838 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:56:49.075821 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b\": container with ID starting with 15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b not found: ID does not exist" containerID="15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b" Sep 09 20:56:49.075892 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.075842 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b"} err="failed to get container status \"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b\": rpc error: code = NotFound desc = could not find container \"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b\": container with ID starting with 15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b not found: ID does not exist" Sep 09 20:56:49.075892 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.075855 2573 scope.go:117] "RemoveContainer" containerID="c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" Sep 09 20:56:49.076145 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:56:49.076127 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a\": container with ID starting with c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a not found: ID does not exist" containerID="c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" Sep 09 20:56:49.076214 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076151 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a"} err="failed to get container status \"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a\": rpc error: code = NotFound desc = could not find container \"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a\": container with ID starting with c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a not found: ID does not exist" Sep 09 20:56:49.076214 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076167 2573 scope.go:117] "RemoveContainer" containerID="9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0" Sep 09 20:56:49.076474 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076442 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0"} err="failed to get container status \"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0\": rpc error: code = NotFound desc = could not find container \"9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0\": container with ID starting with 9b57949923b8bfb5a50bf6f5a0619bc2c4195a081c5306f61965b3acd8cbbdb0 not found: ID does not exist" Sep 09 20:56:49.076474 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076464 2573 scope.go:117] "RemoveContainer" containerID="15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b" Sep 09 20:56:49.076756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076730 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b"} err="failed to get container status \"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b\": rpc error: code = NotFound desc = could not find container \"15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b\": container with ID starting with 15fce7b9551016e38d812880f890be286e8745b225dc055e06dc34477a217b4b not found: ID does not exist" Sep 09 20:56:49.076756 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.076756 2573 scope.go:117] "RemoveContainer" containerID="c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a" Sep 09 20:56:49.077137 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.077115 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a"} err="failed to get container status \"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a\": rpc error: code = NotFound desc = could not find container \"c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a\": container with ID starting with c49ee74492ff7f0d9ff47fce045e4fcf26c7bb2851cb6bff5d9abef01458064a not found: ID does not exist" Sep 09 20:56:49.081744 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:49.081706 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-d744d47cc-jv8ws"] Sep 09 20:56:50.020782 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:50.020740 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" path="/var/lib/kubelet/pods/4af88233-73b0-44df-a363-c00e1fe85faa/volumes" Sep 09 20:56:56.479958 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.479925 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480339 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="llm-d-routing-sidecar" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480351 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="llm-d-routing-sidecar" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480375 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480381 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480389 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="storage-initializer" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480396 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="storage-initializer" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480453 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="llm-d-routing-sidecar" Sep 09 20:56:56.480481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.480463 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af88233-73b0-44df-a363-c00e1fe85faa" containerName="main" Sep 09 20:56:56.485553 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.485530 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.488363 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.488344 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-2gwnc\"" Sep 09 20:56:56.488481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.488454 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\"" Sep 09 20:56:56.489775 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.489754 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:56:56.489879 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.489773 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\"" Sep 09 20:56:56.489879 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.489775 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\"" Sep 09 20:56:56.489879 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.489806 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\"" Sep 09 20:56:56.507588 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.507557 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:56:56.574982 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.574945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.574993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjh8\" (UniqueName: \"kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575128 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575166 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.575377 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.575262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676668 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676668 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjh8\" (UniqueName: \"kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676740 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.676931 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.676852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.677274 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.677111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.677274 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.677172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.677399 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.677271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.677399 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.677379 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.677535 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.677515 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.679213 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.679192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.679510 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.679489 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.688844 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.688817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjh8\" (UniqueName: \"kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8\") pod \"lora-single-adapter-test-kserve-5d867745ff-xw2d6\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:56.795620 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:56.795572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:56:57.111805 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:56:57.111776 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fe846a_9314_4f58_a31e_b3f63aea9dd2.slice/crio-76dcdaf6343b52e5401868f584806677d622f7f331205740e24a32eac271a2ae WatchSource:0}: Error finding container 76dcdaf6343b52e5401868f584806677d622f7f331205740e24a32eac271a2ae: Status 404 returned error can't find the container with id 76dcdaf6343b52e5401868f584806677d622f7f331205740e24a32eac271a2ae Sep 09 20:56:57.130039 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:57.130015 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:56:58.089522 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:58.089484 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerStarted","Data":"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3"} Sep 09 20:56:58.089909 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:56:58.089542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerStarted","Data":"76dcdaf6343b52e5401868f584806677d622f7f331205740e24a32eac271a2ae"} Sep 09 20:57:00.100800 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:00.100764 2573 generic.go:356] "Generic (PLEG): container finished" podID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerID="2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3" exitCode=0 Sep 09 20:57:00.101310 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:00.100821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerDied","Data":"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3"} Sep 09 20:57:01.107814 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:01.107779 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerStarted","Data":"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c"} Sep 09 20:57:01.132736 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:01.132686 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podStartSLOduration=2.390243813 podStartE2EDuration="5.13266825s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.742424437s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:01.132208858 +0000 UTC m=+3707.651088618" watchObservedRunningTime="2026-09-09 20:57:01.13266825 +0000 UTC m=+3707.651547963" Sep 09 20:57:01.795709 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:01.795668 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:57:06.795941 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:06.795892 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:57:06.797415 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:06.797384 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:57:16.796004 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:16.795961 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:57:26.796670 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:26.796621 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:57:36.797051 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:36.797001 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:57:43.314211 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.314173 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 20:57:43.318250 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.318206 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.321140 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321118 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:57:43.321620 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321597 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\"" Sep 09 20:57:43.321730 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321628 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cwr8d\"" Sep 09 20:57:43.321730 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321721 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\"" Sep 09 20:57:43.321864 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321728 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Sep 09 20:57:43.321864 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.321598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\"" Sep 09 20:57:43.330838 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.330813 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 20:57:43.431281 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431281 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kpd\" (UniqueName: \"kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.431718 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.431611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532339 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kpd\" (UniqueName: \"kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532598 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.532930 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.532927 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.533105 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.533071 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.533309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.533251 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.534862 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.534841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.535189 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.535172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.546045 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.546013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kpd\" (UniqueName: \"kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd\") pod \"custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.631041 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.630957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:43.917052 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:43.916962 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 20:57:43.920144 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:57:43.920115 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341c3d10_89b1_47c6_82b0_5e6dfd670995.slice/crio-6d3e3aafd97e9e11b697cd1cccbb9943d707786f2a0740887ffd58f5fafd7d2f WatchSource:0}: Error finding container 6d3e3aafd97e9e11b697cd1cccbb9943d707786f2a0740887ffd58f5fafd7d2f: Status 404 returned error can't find the container with id 6d3e3aafd97e9e11b697cd1cccbb9943d707786f2a0740887ffd58f5fafd7d2f Sep 09 20:57:44.344818 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:44.344783 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerStarted","Data":"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227"} Sep 09 20:57:44.345211 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:44.344849 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerStarted","Data":"6d3e3aafd97e9e11b697cd1cccbb9943d707786f2a0740887ffd58f5fafd7d2f"} Sep 09 20:57:46.796389 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:46.796336 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:57:47.362800 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:47.362766 2573 generic.go:356] "Generic (PLEG): container finished" podID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerID="8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227" exitCode=0 Sep 09 20:57:47.362980 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:47.362805 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerDied","Data":"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227"} Sep 09 20:57:48.370771 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:48.370732 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerStarted","Data":"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c"} Sep 09 20:57:48.395717 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:48.395645 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podStartSLOduration=3.047374016 podStartE2EDuration="5.395622996s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.34824898s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:48.395109379 +0000 UTC m=+3754.913989093" watchObservedRunningTime="2026-09-09 20:57:48.395622996 +0000 UTC m=+3754.914502710" Sep 09 20:57:53.631479 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:53.631433 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:53.631885 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:53.631515 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:57:53.632933 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:53.632906 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:57:56.796400 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:57:56.796349 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:58:03.632251 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:03.632181 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:06.796518 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:06.796421 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:58:13.631938 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:13.631884 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:16.796495 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:16.796443 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:58:23.632299 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:23.632211 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:26.805592 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:26.805557 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:58:26.813430 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:26.813393 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:58:33.632381 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:33.632337 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:43.632467 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:43.632419 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:53.632427 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:53.632365 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:58:56.248734 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:56.248708 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5d867745ff-xw2d6_58fe846a-9314-4f58-a31e-b3f63aea9dd2/storage-initializer/0.log" Sep 09 20:58:56.259216 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:56.259193 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5d867745ff-xw2d6_58fe846a-9314-4f58-a31e-b3f63aea9dd2/main/0.log" Sep 09 20:58:58.316970 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:58:58.316926 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:59:03.631776 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:03.631728 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:59:13.340954 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:13.340893 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" containerID="cri-o://85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c" gracePeriod=45 Sep 09 20:59:13.631803 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:13.631699 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:59:13.806958 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:13.806919 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 09 20:59:14.407332 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.407303 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:59:14.540095 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540138 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540271 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540309 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540299 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540365 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cjh8\" (UniqueName: \"kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540394 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540551 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540444 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs\") pod \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\" (UID: \"58fe846a-9314-4f58-a31e-b3f63aea9dd2\") " Sep 09 20:59:14.540751 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540435 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:59:14.540889 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540538 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:59:14.540889 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.540860 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home" (OuterVolumeSpecName: "home") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:59:14.543108 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.543074 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8" (OuterVolumeSpecName: "kube-api-access-8cjh8") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "kube-api-access-8cjh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:59:14.543255 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.543164 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm" (OuterVolumeSpecName: "dshm") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:59:14.543326 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.543278 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:59:14.552459 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.552411 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:59:14.566717 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.566672 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58fe846a-9314-4f58-a31e-b3f63aea9dd2" (UID: "58fe846a-9314-4f58-a31e-b3f63aea9dd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:59:14.642160 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642116 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642160 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642150 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642160 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642159 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjh8\" (UniqueName: \"kubernetes.io/projected/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kube-api-access-8cjh8\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642160 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642169 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642479 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642181 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642479 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642190 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58fe846a-9314-4f58-a31e-b3f63aea9dd2-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642479 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642197 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.642479 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.642204 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fe846a-9314-4f58-a31e-b3f63aea9dd2-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 20:59:14.871808 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.871719 2573 generic.go:356] "Generic (PLEG): container finished" podID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerID="85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c" exitCode=0 Sep 09 20:59:14.871808 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.871767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerDied","Data":"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c"} Sep 09 20:59:14.872129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.871822 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" event={"ID":"58fe846a-9314-4f58-a31e-b3f63aea9dd2","Type":"ContainerDied","Data":"76dcdaf6343b52e5401868f584806677d622f7f331205740e24a32eac271a2ae"} Sep 09 20:59:14.872129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.871832 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6" Sep 09 20:59:14.872129 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.871851 2573 scope.go:117] "RemoveContainer" containerID="85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c" Sep 09 20:59:14.881410 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.881385 2573 scope.go:117] "RemoveContainer" containerID="2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3" Sep 09 20:59:14.890662 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.890640 2573 scope.go:117] "RemoveContainer" containerID="85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c" Sep 09 20:59:14.891000 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:59:14.890977 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c\": container with ID starting with 85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c not found: ID does not exist" containerID="85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c" Sep 09 20:59:14.891054 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.891010 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c"} err="failed to get container status \"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c\": rpc error: code = NotFound desc = could not find container \"85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c\": container with ID starting with 85899810044dd5acaedd1822550a9b11896a64af650d77e322d63ec9e4445f9c not found: ID does not exist" Sep 09 20:59:14.891054 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.891030 2573 scope.go:117] "RemoveContainer" containerID="2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3" Sep 09 20:59:14.891340 ip-10-0-141-196 kubenswrapper[2573]: E0909 20:59:14.891317 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3\": container with ID starting with 2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3 not found: ID does not exist" containerID="2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3" Sep 09 20:59:14.891390 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.891351 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3"} err="failed to get container status \"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3\": rpc error: code = NotFound desc = could not find container \"2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3\": container with ID starting with 2f80e5323e1c78f0c3a413256668fbbf90a62840abfb76b151bfb089e4655ad3 not found: ID does not exist" Sep 09 20:59:14.898415 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.898386 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:59:14.902614 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:14.902589 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d867745ff-xw2d6"] Sep 09 20:59:16.021465 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:16.021432 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" path="/var/lib/kubelet/pods/58fe846a-9314-4f58-a31e-b3f63aea9dd2/volumes" Sep 09 20:59:23.631547 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:23.631502 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:59:33.631987 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:33.631941 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:59:37.745933 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.745850 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 20:59:37.746486 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.746465 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" Sep 09 20:59:37.746538 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.746491 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" Sep 09 20:59:37.746538 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.746512 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="storage-initializer" Sep 09 20:59:37.746538 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.746521 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="storage-initializer" Sep 09 20:59:37.746644 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.746616 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="58fe846a-9314-4f58-a31e-b3f63aea9dd2" containerName="main" Sep 09 20:59:37.750915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.750893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.754383 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.754357 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\"" Sep 09 20:59:37.754893 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.754875 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 20:59:37.754975 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.754945 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\"" Sep 09 20:59:37.755539 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.755522 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\"" Sep 09 20:59:37.757742 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.757721 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\"" Sep 09 20:59:37.762528 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.762510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-2hrr8\"" Sep 09 20:59:37.777186 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.777155 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 20:59:37.851679 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851679 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851785 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtrk\" (UniqueName: \"kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.851915 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.852098 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.851942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.952903 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.952858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.952903 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.952902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.952953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.952992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtrk\" (UniqueName: \"kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953436 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953481 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953652 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.953652 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.953629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.954120 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.954092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.955443 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.955412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.955965 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.955941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:37.970267 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:37.970210 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtrk\" (UniqueName: \"kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk\") pod \"lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:38.062922 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:38.062881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:38.353478 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:38.353449 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 20:59:38.356518 ip-10-0-141-196 kubenswrapper[2573]: W0909 20:59:38.356491 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210ddcfe_b20c_40b2_914d_0ee37ca23564.slice/crio-81e1d170d0ed97c5705aa3a9572a03c7300c7f1d0687ba2f880acce9c92e0c48 WatchSource:0}: Error finding container 81e1d170d0ed97c5705aa3a9572a03c7300c7f1d0687ba2f880acce9c92e0c48: Status 404 returned error can't find the container with id 81e1d170d0ed97c5705aa3a9572a03c7300c7f1d0687ba2f880acce9c92e0c48 Sep 09 20:59:39.013244 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:39.013192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerStarted","Data":"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2"} Sep 09 20:59:39.013763 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:39.013274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerStarted","Data":"81e1d170d0ed97c5705aa3a9572a03c7300c7f1d0687ba2f880acce9c92e0c48"} Sep 09 20:59:42.031431 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:42.031392 2573 generic.go:356] "Generic (PLEG): container finished" podID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerID="fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2" exitCode=0 Sep 09 20:59:42.032002 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:42.031464 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerDied","Data":"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2"} Sep 09 20:59:43.038562 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:43.038526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerStarted","Data":"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702"} Sep 09 20:59:43.063029 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:43.062987 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:43.063240 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:43.062975 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podStartSLOduration=2.984018403 podStartE2EDuration="6.062953402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.078934999s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:43.060599159 +0000 UTC m=+3869.579478871" watchObservedRunningTime="2026-09-09 20:59:43.062953402 +0000 UTC m=+3869.581833114" Sep 09 20:59:43.632162 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:43.632121 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused" Sep 09 20:59:48.063955 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:48.063918 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 20:59:48.065894 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:48.065859 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 20:59:53.641830 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:53.641788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:59:53.650023 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:53.649990 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 20:59:58.064133 ip-10-0-141-196 kubenswrapper[2573]: I0909 20:59:58.064076 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:03.210779 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:03.210730 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 21:00:08.064147 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:08.064088 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:18.063712 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:18.063665 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:18.234463 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:18.234403 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" containerID="cri-o://35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c" gracePeriod=45 Sep 09 21:00:18.997303 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:18.997274 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 21:00:19.062554 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5kpd\" (UniqueName: \"kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062580 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062626 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062652 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062719 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.062764 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.062746 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.063156 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.063124 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:00:19.063588 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.063555 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache" (OuterVolumeSpecName: "model-cache") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:00:19.063700 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.063610 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home" (OuterVolumeSpecName: "home") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:00:19.063700 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.063664 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location\") pod \"341c3d10-89b1-47c6-82b0-5e6dfd670995\" (UID: \"341c3d10-89b1-47c6-82b0-5e6dfd670995\") " Sep 09 21:00:19.064142 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.064034 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.064142 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.064051 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/341c3d10-89b1-47c6-82b0-5e6dfd670995-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.064142 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.064065 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.065638 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.065608 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd" (OuterVolumeSpecName: "kube-api-access-d5kpd") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "kube-api-access-d5kpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:00:19.065746 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.065641 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm" (OuterVolumeSpecName: "dshm") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:00:19.065830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.065807 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:00:19.078967 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.078929 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:00:19.093353 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.093307 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "341c3d10-89b1-47c6-82b0-5e6dfd670995" (UID: "341c3d10-89b1-47c6-82b0-5e6dfd670995"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:00:19.165442 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.165390 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/341c3d10-89b1-47c6-82b0-5e6dfd670995-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.165442 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.165430 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.165442 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.165440 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kpd\" (UniqueName: \"kubernetes.io/projected/341c3d10-89b1-47c6-82b0-5e6dfd670995-kube-api-access-d5kpd\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.165442 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.165448 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.165442 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.165457 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/341c3d10-89b1-47c6-82b0-5e6dfd670995-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:00:19.237630 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.237587 2573 generic.go:356] "Generic (PLEG): container finished" podID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerID="35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c" exitCode=0 Sep 09 21:00:19.237831 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.237639 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" Sep 09 21:00:19.237831 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.237638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerDied","Data":"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c"} Sep 09 21:00:19.237831 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.237761 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw" event={"ID":"341c3d10-89b1-47c6-82b0-5e6dfd670995","Type":"ContainerDied","Data":"6d3e3aafd97e9e11b697cd1cccbb9943d707786f2a0740887ffd58f5fafd7d2f"} Sep 09 21:00:19.237831 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.237784 2573 scope.go:117] "RemoveContainer" containerID="35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c" Sep 09 21:00:19.247754 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.247726 2573 scope.go:117] "RemoveContainer" containerID="8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227" Sep 09 21:00:19.258787 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.258376 2573 scope.go:117] "RemoveContainer" containerID="35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c" Sep 09 21:00:19.259072 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:00:19.259042 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c\": container with ID starting with 35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c not found: ID does not exist" containerID="35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c" Sep 09 21:00:19.259179 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.259084 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c"} err="failed to get container status \"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c\": rpc error: code = NotFound desc = could not find container \"35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c\": container with ID starting with 35934ca34b83729922e584a2282ed9554573bc2374e3514d6e421603b180026c not found: ID does not exist" Sep 09 21:00:19.259179 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.259102 2573 scope.go:117] "RemoveContainer" containerID="8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227" Sep 09 21:00:19.259859 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:00:19.259838 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227\": container with ID starting with 8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227 not found: ID does not exist" containerID="8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227" Sep 09 21:00:19.259925 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.259866 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227"} err="failed to get container status \"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227\": rpc error: code = NotFound desc = could not find container \"8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227\": container with ID starting with 8f58688bc74949d31c3452c1cf308b2b0cb7ae27feae25099299ba9e7a0fa227 not found: ID does not exist" Sep 09 21:00:19.273307 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.273278 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 21:00:19.279000 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:19.278975 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-d9b8cf867-gb2qw"] Sep 09 21:00:20.015408 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:20.015375 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" path="/var/lib/kubelet/pods/341c3d10-89b1-47c6-82b0-5e6dfd670995/volumes" Sep 09 21:00:28.063624 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:28.063576 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:38.064301 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:38.064256 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:48.063374 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:48.063324 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:00:58.064025 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:00:58.063980 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:01:08.074085 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:08.074051 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 21:01:08.082515 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:08.082485 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 21:01:40.668667 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:40.668639 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s_210ddcfe-b20c-40b2-914d-0ee37ca23564/storage-initializer/0.log" Sep 09 21:01:40.678435 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:40.678413 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s_210ddcfe-b20c-40b2-914d-0ee37ca23564/main/0.log" Sep 09 21:01:41.742511 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:41.742475 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 21:01:56.765563 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:56.765461 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" containerID="cri-o://c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702" gracePeriod=45 Sep 09 21:01:57.074721 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.074680 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused" Sep 09 21:01:57.423950 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.423923 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 21:01:57.595701 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595603 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.595701 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595654 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.595701 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595686 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.595701 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595703 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qtrk\" (UniqueName: \"kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.596067 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595721 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.596067 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595772 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.596067 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595790 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.596067 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.595859 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert\") pod \"210ddcfe-b20c-40b2-914d-0ee37ca23564\" (UID: \"210ddcfe-b20c-40b2-914d-0ee37ca23564\") " Sep 09 21:01:57.596067 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.596058 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache" (OuterVolumeSpecName: "model-cache") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:01:57.596391 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.596161 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.596391 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.596165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home" (OuterVolumeSpecName: "home") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:01:57.596523 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.596441 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:01:57.598484 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.598445 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:01:57.598601 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.598520 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm" (OuterVolumeSpecName: "dshm") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:01:57.598601 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.598565 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk" (OuterVolumeSpecName: "kube-api-access-5qtrk") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "kube-api-access-5qtrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:01:57.613861 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.613798 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:01:57.621971 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.621912 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "210ddcfe-b20c-40b2-914d-0ee37ca23564" (UID: "210ddcfe-b20c-40b2-914d-0ee37ca23564"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:01:57.697578 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697538 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697578 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697571 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/210ddcfe-b20c-40b2-914d-0ee37ca23564-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697801 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697603 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697801 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697611 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697801 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697620 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/210ddcfe-b20c-40b2-914d-0ee37ca23564-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697801 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697628 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtrk\" (UniqueName: \"kubernetes.io/projected/210ddcfe-b20c-40b2-914d-0ee37ca23564-kube-api-access-5qtrk\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.697801 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.697637 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210ddcfe-b20c-40b2-914d-0ee37ca23564-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:01:57.762534 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.762502 2573 generic.go:356] "Generic (PLEG): container finished" podID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerID="c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702" exitCode=0 Sep 09 21:01:57.762740 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.762565 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" Sep 09 21:01:57.762740 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.762592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerDied","Data":"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702"} Sep 09 21:01:57.762740 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.762650 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s" event={"ID":"210ddcfe-b20c-40b2-914d-0ee37ca23564","Type":"ContainerDied","Data":"81e1d170d0ed97c5705aa3a9572a03c7300c7f1d0687ba2f880acce9c92e0c48"} Sep 09 21:01:57.762740 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.762674 2573 scope.go:117] "RemoveContainer" containerID="c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702" Sep 09 21:01:57.772471 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.772291 2573 scope.go:117] "RemoveContainer" containerID="fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2" Sep 09 21:01:57.781256 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.781218 2573 scope.go:117] "RemoveContainer" containerID="c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702" Sep 09 21:01:57.781573 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:01:57.781552 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702\": container with ID starting with c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702 not found: ID does not exist" containerID="c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702" Sep 09 21:01:57.781653 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.781588 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702"} err="failed to get container status \"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702\": rpc error: code = NotFound desc = could not find container \"c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702\": container with ID starting with c6cfb2067758936f8d7cd107c2a84885242d32d8e335067f767721abdf9ef702 not found: ID does not exist" Sep 09 21:01:57.781653 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.781614 2573 scope.go:117] "RemoveContainer" containerID="fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2" Sep 09 21:01:57.781909 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:01:57.781890 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2\": container with ID starting with fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2 not found: ID does not exist" containerID="fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2" Sep 09 21:01:57.781958 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.781915 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2"} err="failed to get container status \"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2\": rpc error: code = NotFound desc = could not find container \"fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2\": container with ID starting with fa493b8bacbb68fcf1470eb3d0c19ecc8d95642ca5a508c3906942e7fa8468f2 not found: ID does not exist" Sep 09 21:01:57.792331 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.792300 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 21:01:57.799854 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:57.799818 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57cbfb6956-rvd8s"] Sep 09 21:01:58.017061 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:01:58.016977 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" path="/var/lib/kubelet/pods/210ddcfe-b20c-40b2-914d-0ee37ca23564/volumes" Sep 09 21:02:06.970693 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.970649 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:02:06.971243 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971204 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="storage-initializer" Sep 09 21:02:06.971243 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971241 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="storage-initializer" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971258 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971268 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971302 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971310 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971322 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="storage-initializer" Sep 09 21:02:06.971337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971330 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="storage-initializer" Sep 09 21:02:06.971531 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971417 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="341c3d10-89b1-47c6-82b0-5e6dfd670995" containerName="main" Sep 09 21:02:06.971531 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.971430 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="210ddcfe-b20c-40b2-914d-0ee37ca23564" containerName="main" Sep 09 21:02:06.974012 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.973991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:06.976605 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.976584 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\"" Sep 09 21:02:06.976761 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.976735 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\"" Sep 09 21:02:06.977078 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.977062 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\"" Sep 09 21:02:06.977327 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.977309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-rvvb9\"" Sep 09 21:02:06.977439 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.977362 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\"" Sep 09 21:02:06.977810 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.977784 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 21:02:06.988956 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:06.988932 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:02:07.084487 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084692 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084692 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084692 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084875 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084875 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084875 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084774 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.084875 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.084800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89gz\" (UniqueName: \"kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185643 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89gz\" (UniqueName: \"kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.185830 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.185781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.186245 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.186122 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.186245 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.186193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.186369 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.186270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.186369 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.186283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.186467 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.186449 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.188080 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.188062 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.188529 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.188511 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.195098 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.195068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89gz\" (UniqueName: \"kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz\") pod \"tls-verification-test-kserve-c5bd6cb78-vvvr6\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.286721 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.286690 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:07.565739 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.565659 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:02:07.569340 ip-10-0-141-196 kubenswrapper[2573]: W0909 21:02:07.569312 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc41a33a_6c37_4826_9512_f2f4d86b22c2.slice/crio-8e9641b8d80b6194d27c66ae799877d35baa6c0524827c929323662186ac23d9 WatchSource:0}: Error finding container 8e9641b8d80b6194d27c66ae799877d35baa6c0524827c929323662186ac23d9: Status 404 returned error can't find the container with id 8e9641b8d80b6194d27c66ae799877d35baa6c0524827c929323662186ac23d9 Sep 09 21:02:07.814799 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.814760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerStarted","Data":"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f"} Sep 09 21:02:07.814971 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:07.814824 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerStarted","Data":"8e9641b8d80b6194d27c66ae799877d35baa6c0524827c929323662186ac23d9"} Sep 09 21:02:10.833623 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:10.833594 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerID="d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f" exitCode=0 Sep 09 21:02:10.834027 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:10.833678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerDied","Data":"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f"} Sep 09 21:02:11.840540 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:11.840502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerStarted","Data":"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0"} Sep 09 21:02:11.872085 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:11.866531 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podStartSLOduration=3.617032932 podStartE2EDuration="5.866509973s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.249477041s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:11.863065284 +0000 UTC m=+4018.381944996" watchObservedRunningTime="2026-09-09 21:02:11.866509973 +0000 UTC m=+4018.385389687" Sep 09 21:02:12.287536 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:12.287502 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:17.287113 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:17.287066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:02:17.288762 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:17.288730 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:02:27.288100 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:27.288054 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:02:37.287546 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:37.287373 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:02:47.287337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:47.287288 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:02:57.287422 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:02:57.287371 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:07.287119 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:07.287071 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:17.287149 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:17.287101 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:27.287577 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:27.287535 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:37.287299 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:37.287212 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:47.287700 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:47.287650 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:03:57.287626 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:03:57.287581 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:04:07.287751 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:07.287645 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:04:17.298676 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:17.298636 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:04:17.307143 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:17.307111 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:04:22.619044 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:22.619006 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:04:37.643435 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:37.643369 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" containerID="cri-o://7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0" gracePeriod=45 Sep 09 21:04:38.299766 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.299713 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused" Sep 09 21:04:38.494700 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.494673 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:04:38.607938 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.607896 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.607938 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.607942 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608166 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608022 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c89gz\" (UniqueName: \"kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608166 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608072 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608166 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608143 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608368 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608368 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608296 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608368 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608333 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs\") pod \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\" (UID: \"fc41a33a-6c37-4826-9512-f2f4d86b22c2\") " Sep 09 21:04:38.608533 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608447 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:04:38.608735 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608698 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc41a33a-6c37-4826-9512-f2f4d86b22c2-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.608863 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home" (OuterVolumeSpecName: "home") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:04:38.608863 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.608837 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:04:38.611143 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.611115 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz" (OuterVolumeSpecName: "kube-api-access-c89gz") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "kube-api-access-c89gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:04:38.611291 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.611155 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm" (OuterVolumeSpecName: "dshm") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:04:38.611291 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.611163 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:04:38.619250 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.619192 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerID="7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0" exitCode=0 Sep 09 21:04:38.619370 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.619271 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" Sep 09 21:04:38.619370 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.619271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerDied","Data":"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0"} Sep 09 21:04:38.619370 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.619345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6" event={"ID":"fc41a33a-6c37-4826-9512-f2f4d86b22c2","Type":"ContainerDied","Data":"8e9641b8d80b6194d27c66ae799877d35baa6c0524827c929323662186ac23d9"} Sep 09 21:04:38.619483 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.619376 2573 scope.go:117] "RemoveContainer" containerID="7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0" Sep 09 21:04:38.623647 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.623617 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:04:38.629839 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.629816 2573 scope.go:117] "RemoveContainer" containerID="d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f" Sep 09 21:04:38.638402 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.638262 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc41a33a-6c37-4826-9512-f2f4d86b22c2" (UID: "fc41a33a-6c37-4826-9512-f2f4d86b22c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:04:38.638478 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.638449 2573 scope.go:117] "RemoveContainer" containerID="7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0" Sep 09 21:04:38.638809 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:04:38.638786 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0\": container with ID starting with 7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0 not found: ID does not exist" containerID="7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0" Sep 09 21:04:38.638858 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.638821 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0"} err="failed to get container status \"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0\": rpc error: code = NotFound desc = could not find container \"7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0\": container with ID starting with 7bce56a603221f7f860fd0140d7547698986ec2c58191cdf96dfc1b727b613a0 not found: ID does not exist" Sep 09 21:04:38.638858 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.638841 2573 scope.go:117] "RemoveContainer" containerID="d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f" Sep 09 21:04:38.639096 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:04:38.639081 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f\": container with ID starting with d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f not found: ID does not exist" containerID="d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f" Sep 09 21:04:38.639145 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.639099 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f"} err="failed to get container status \"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f\": rpc error: code = NotFound desc = could not find container \"d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f\": container with ID starting with d0f1b1c9c1b7c2bfe874dbee46e20b0b5e1fc242ef93a7000362fcf43a2f3d5f not found: ID does not exist" Sep 09 21:04:38.709903 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709859 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.709903 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709895 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.709903 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709908 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc41a33a-6c37-4826-9512-f2f4d86b22c2-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.710420 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709919 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.710420 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709930 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89gz\" (UniqueName: \"kubernetes.io/projected/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kube-api-access-c89gz\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.710420 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709942 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.710420 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.709953 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc41a33a-6c37-4826-9512-f2f4d86b22c2-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:04:38.942045 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.942015 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:04:38.948684 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:38.948650 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c5bd6cb78-vvvr6"] Sep 09 21:04:40.014889 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:40.014857 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" path="/var/lib/kubelet/pods/fc41a33a-6c37-4826-9512-f2f4d86b22c2/volumes" Sep 09 21:04:52.129773 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.129739 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:04:52.130160 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.130131 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="storage-initializer" Sep 09 21:04:52.130160 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.130141 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="storage-initializer" Sep 09 21:04:52.130160 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.130151 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" Sep 09 21:04:52.130160 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.130157 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" Sep 09 21:04:52.130307 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.130259 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc41a33a-6c37-4826-9512-f2f4d86b22c2" containerName="main" Sep 09 21:04:52.135097 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.135072 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.137861 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.137828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 21:04:52.138029 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.138008 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\"" Sep 09 21:04:52.138116 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.137909 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4wf97\"" Sep 09 21:04:52.138193 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.137952 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\"" Sep 09 21:04:52.138193 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.137989 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\"" Sep 09 21:04:52.138326 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.137885 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\"" Sep 09 21:04:52.148504 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.148475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:04:52.239337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239337 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcxm\" (UniqueName: \"kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.239579 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.239504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.340674 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.340638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.340674 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.340672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.340947 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.340700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341011 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.340991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341072 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341129 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcxm\" (UniqueName: \"kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341182 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341133 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341182 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341373 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341373 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341503 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.341562 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.341512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.342079 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.342055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.343585 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.343554 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.344259 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.344206 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.355691 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.355655 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcxm\" (UniqueName: \"kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm\") pod \"prestop-hook-test-kserve-847f84995d-d54wf\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.446857 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.446764 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:04:52.731549 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:52.731520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:04:52.734165 ip-10-0-141-196 kubenswrapper[2573]: W0909 21:04:52.734134 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dc884c_b192_4a28_ac94_e13b9acefcba.slice/crio-7e8f87bd581ab598d5f6e7cf10291639000b6e0b938655fc09f65879450d6818 WatchSource:0}: Error finding container 7e8f87bd581ab598d5f6e7cf10291639000b6e0b938655fc09f65879450d6818: Status 404 returned error can't find the container with id 7e8f87bd581ab598d5f6e7cf10291639000b6e0b938655fc09f65879450d6818 Sep 09 21:04:53.696563 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:53.696517 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerStarted","Data":"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e"} Sep 09 21:04:53.696956 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:53.696588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerStarted","Data":"7e8f87bd581ab598d5f6e7cf10291639000b6e0b938655fc09f65879450d6818"} Sep 09 21:04:55.708438 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:55.708397 2573 generic.go:356] "Generic (PLEG): container finished" podID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerID="60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e" exitCode=0 Sep 09 21:04:55.708885 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:55.708434 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerDied","Data":"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e"} Sep 09 21:04:56.715767 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:56.715736 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerStarted","Data":"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd"} Sep 09 21:04:56.746634 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:56.746557 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podStartSLOduration=2.45028377 podStartE2EDuration="4.746529495s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296245725s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:56.744262471 +0000 UTC m=+4183.263142184" watchObservedRunningTime="2026-09-09 21:04:56.746529495 +0000 UTC m=+4183.265409268" Sep 09 21:04:57.447072 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:04:57.447033 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:05:02.447448 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:02.447396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:05:02.448817 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:02.448780 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:05:12.447393 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:12.447341 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:05:22.447931 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:22.447882 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:05:32.447975 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:32.447925 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:05:42.447340 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:42.447298 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:05:52.448093 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:05:52.448031 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:02.447688 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:02.447638 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:12.448026 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:12.447972 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:22.447797 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:22.447732 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:32.447450 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:32.447399 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:42.448185 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:42.448137 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:06:52.447543 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:06:52.447482 2573 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:07:02.458218 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:02.458182 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:07:02.466070 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:02.466047 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:07:02.578010 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:02.577979 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:07:19.408130 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:19.407998 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" containerID="cri-o://cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd" gracePeriod=45 Sep 09 21:07:19.459522 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:19.459477 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused" Sep 09 21:07:20.165930 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.165906 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:07:20.259730 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259646 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.259730 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259687 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.259730 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259728 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260029 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259756 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260029 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259772 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260029 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259802 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260029 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.259820 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260301 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.260032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djcxm\" (UniqueName: \"kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm\") pod \"71dc884c-b192-4a28-ac94-e13b9acefcba\" (UID: \"71dc884c-b192-4a28-ac94-e13b9acefcba\") " Sep 09 21:07:20.260301 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.260056 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache" (OuterVolumeSpecName: "model-cache") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:20.260417 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.260351 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:07:20.260806 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.260776 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home" (OuterVolumeSpecName: "home") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:20.261193 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.261172 2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71dc884c-b192-4a28-ac94-e13b9acefcba-cabundle-cert\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.261343 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.261199 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-model-cache\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.261343 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.261213 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-home\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.262849 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.262814 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm" (OuterVolumeSpecName: "dshm") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:20.262970 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.262884 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm" (OuterVolumeSpecName: "kube-api-access-djcxm") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "kube-api-access-djcxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:07:20.263033 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.262979 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:07:20.273285 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.273209 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:20.285874 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.285827 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71dc884c-b192-4a28-ac94-e13b9acefcba" (UID: "71dc884c-b192-4a28-ac94-e13b9acefcba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:20.362713 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.362677 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-tmp-dir\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.362713 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.362708 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-dshm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.362713 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.362716 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71dc884c-b192-4a28-ac94-e13b9acefcba-tls-certs\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.362713 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.362725 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcxm\" (UniqueName: \"kubernetes.io/projected/71dc884c-b192-4a28-ac94-e13b9acefcba-kube-api-access-djcxm\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.363040 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.362733 2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71dc884c-b192-4a28-ac94-e13b9acefcba-kserve-provision-location\") on node \"ip-10-0-141-196.ec2.internal\" DevicePath \"\"" Sep 09 21:07:20.462945 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.462908 2573 generic.go:356] "Generic (PLEG): container finished" podID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerID="cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd" exitCode=0 Sep 09 21:07:20.463408 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.462968 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" Sep 09 21:07:20.463408 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.462978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerDied","Data":"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd"} Sep 09 21:07:20.463408 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.463024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf" event={"ID":"71dc884c-b192-4a28-ac94-e13b9acefcba","Type":"ContainerDied","Data":"7e8f87bd581ab598d5f6e7cf10291639000b6e0b938655fc09f65879450d6818"} Sep 09 21:07:20.463408 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.463046 2573 scope.go:117] "RemoveContainer" containerID="cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd" Sep 09 21:07:20.472994 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.472973 2573 scope.go:117] "RemoveContainer" containerID="60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e" Sep 09 21:07:20.481252 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.481206 2573 scope.go:117] "RemoveContainer" containerID="cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd" Sep 09 21:07:20.481568 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:07:20.481548 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd\": container with ID starting with cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd not found: ID does not exist" containerID="cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd" Sep 09 21:07:20.481640 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.481575 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd"} err="failed to get container status \"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd\": rpc error: code = NotFound desc = could not find container \"cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd\": container with ID starting with cfb8262503526af2f9f0eb222bba53592bc28a03d3b513e63c0e90c7a95705bd not found: ID does not exist" Sep 09 21:07:20.481640 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.481592 2573 scope.go:117] "RemoveContainer" containerID="60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e" Sep 09 21:07:20.481877 ip-10-0-141-196 kubenswrapper[2573]: E0909 21:07:20.481859 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e\": container with ID starting with 60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e not found: ID does not exist" containerID="60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e" Sep 09 21:07:20.481916 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.481885 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e"} err="failed to get container status \"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e\": rpc error: code = NotFound desc = could not find container \"60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e\": container with ID starting with 60f21f70a06ff1f62b67ab380bb2f25a023d26f64a7b9925ed59da483b1d628e not found: ID does not exist" Sep 09 21:07:20.488426 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.488399 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:07:20.493350 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:20.493324 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-d54wf"] Sep 09 21:07:22.015300 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:07:22.015268 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" path="/var/lib/kubelet/pods/71dc884c-b192-4a28-ac94-e13b9acefcba/volumes" Sep 09 21:08:58.025410 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:08:58.025377 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-5rnjx_98f942f6-35a2-42df-97fe-698ccf44bc2a/manager/0.log" Sep 09 21:08:58.218854 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:08:58.218817 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sksm8_60baa3dd-8dbb-4e9c-812e-955872f82cd3/limitador/0.log" Sep 09 21:08:58.238539 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:08:58.238510 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-28ctb_bb4ded8d-d639-4cf7-b784-2236cd708580/manager/0.log" Sep 09 21:09:04.371807 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.371768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4ljz/must-gather-zfvzt"] Sep 09 21:09:04.372205 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.372152 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" Sep 09 21:09:04.372205 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.372164 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" Sep 09 21:09:04.372205 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.372172 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="storage-initializer" Sep 09 21:09:04.372205 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.372177 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="storage-initializer" Sep 09 21:09:04.372377 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.372265 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dc884c-b192-4a28-ac94-e13b9acefcba" containerName="main" Sep 09 21:09:04.375364 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.375340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.377584 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.377565 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"openshift-service-ca.crt\"" Sep 09 21:09:04.377694 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.377622 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4ljz\"/\"default-dockercfg-mjs97\"" Sep 09 21:09:04.378361 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.378343 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"kube-root-ca.crt\"" Sep 09 21:09:04.386637 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.386611 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/must-gather-zfvzt"] Sep 09 21:09:04.398630 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.398602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpls\" (UniqueName: \"kubernetes.io/projected/32db1742-7f64-41ad-ac12-494dc7f01d9c-kube-api-access-vrpls\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.398791 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.398648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32db1742-7f64-41ad-ac12-494dc7f01d9c-must-gather-output\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.499620 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.499586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpls\" (UniqueName: \"kubernetes.io/projected/32db1742-7f64-41ad-ac12-494dc7f01d9c-kube-api-access-vrpls\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.499620 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.499624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32db1742-7f64-41ad-ac12-494dc7f01d9c-must-gather-output\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.499939 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.499923 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/32db1742-7f64-41ad-ac12-494dc7f01d9c-must-gather-output\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.508714 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.508689 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpls\" (UniqueName: \"kubernetes.io/projected/32db1742-7f64-41ad-ac12-494dc7f01d9c-kube-api-access-vrpls\") pod \"must-gather-zfvzt\" (UID: \"32db1742-7f64-41ad-ac12-494dc7f01d9c\") " pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.684902 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.684795 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" Sep 09 21:09:04.960889 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.960857 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/must-gather-zfvzt"] Sep 09 21:09:04.967191 ip-10-0-141-196 kubenswrapper[2573]: W0909 21:09:04.967165 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32db1742_7f64_41ad_ac12_494dc7f01d9c.slice/crio-40d1e5b316338776b4aece4b7449beff434de1799f2f45c757492c837e641629 WatchSource:0}: Error finding container 40d1e5b316338776b4aece4b7449beff434de1799f2f45c757492c837e641629: Status 404 returned error can't find the container with id 40d1e5b316338776b4aece4b7449beff434de1799f2f45c757492c837e641629 Sep 09 21:09:04.969097 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.969081 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:09:04.980358 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:04.980332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" event={"ID":"32db1742-7f64-41ad-ac12-494dc7f01d9c","Type":"ContainerStarted","Data":"40d1e5b316338776b4aece4b7449beff434de1799f2f45c757492c837e641629"} Sep 09 21:09:05.987505 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:05.987412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" event={"ID":"32db1742-7f64-41ad-ac12-494dc7f01d9c","Type":"ContainerStarted","Data":"366b93c2b97b5169b407497ade83a3809d4e5ee5fecadc4690e01c57adaf6c2f"} Sep 09 21:09:05.987505 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:05.987460 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" event={"ID":"32db1742-7f64-41ad-ac12-494dc7f01d9c","Type":"ContainerStarted","Data":"a8bebea1f611ef514f293e5de755bc676bfdfda4175ae4225daf1f1dd7eb4e62"} Sep 09 21:09:06.005323 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:06.005258 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4ljz/must-gather-zfvzt" podStartSLOduration=1.261608717 podStartE2EDuration="2.00523969s" totalImagesPullingTime="743.630973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:06.00376843 +0000 UTC m=+4432.522648202" watchObservedRunningTime="2026-09-09 21:09:06.00523969 +0000 UTC m=+4432.524119397" Sep 09 21:09:07.470093 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:07.470055 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wdk27_0cd560bd-2201-4588-a031-5c4ab028f42f/global-pull-secret-syncer/0.log" Sep 09 21:09:07.625960 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:07.625920 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w2jsk_72392e60-cc8c-4b7c-80cb-48664d636e08/konnectivity-agent/0.log" Sep 09 21:09:07.740638 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:07.740546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-196.ec2.internal_538be1d961f54997392b43e39784ee03/haproxy/0.log" Sep 09 21:09:13.023369 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:13.023322 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-5rnjx_98f942f6-35a2-42df-97fe-698ccf44bc2a/manager/0.log" Sep 09 21:09:13.255657 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:13.255623 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sksm8_60baa3dd-8dbb-4e9c-812e-955872f82cd3/limitador/0.log" Sep 09 21:09:13.295679 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:13.295623 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-28ctb_bb4ded8d-d639-4cf7-b784-2236cd708580/manager/0.log" Sep 09 21:09:14.705037 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.705000 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/alertmanager/0.log" Sep 09 21:09:14.732271 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.732217 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/config-reloader/0.log" Sep 09 21:09:14.763564 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.763534 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/kube-rbac-proxy-web/0.log" Sep 09 21:09:14.788606 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.788533 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/kube-rbac-proxy/0.log" Sep 09 21:09:14.816678 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.816643 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/kube-rbac-proxy-metric/0.log" Sep 09 21:09:14.846288 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.846211 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/prom-label-proxy/0.log" Sep 09 21:09:14.881246 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.881188 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1eb6610-24ba-4fc7-96d3-a46e8a2b71d6/init-config-reloader/0.log" Sep 09 21:09:14.956456 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.956377 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-tcp66_7d34d8f4-fd11-4824-91fc-9b7784d60775/kube-state-metrics/0.log" Sep 09 21:09:14.979370 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:14.979340 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-tcp66_7d34d8f4-fd11-4824-91fc-9b7784d60775/kube-rbac-proxy-main/0.log" Sep 09 21:09:15.016553 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.016524 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-tcp66_7d34d8f4-fd11-4824-91fc-9b7784d60775/kube-rbac-proxy-self/0.log" Sep 09 21:09:15.067639 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.067611 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f6fc56ccd-knjvx_866f49e7-56f9-420e-b5c4-a2a3d5f4e5ae/metrics-server/0.log" Sep 09 21:09:15.114249 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.114087 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-6q6wc_3eafc3bf-71c1-43e6-9072-1cd31b92db57/monitoring-plugin/0.log" Sep 09 21:09:15.439045 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.438993 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7hv4_57a1a81e-185f-45c0-b337-947b78db271a/node-exporter/0.log" Sep 09 21:09:15.488670 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.488624 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7hv4_57a1a81e-185f-45c0-b337-947b78db271a/kube-rbac-proxy/0.log" Sep 09 21:09:15.535597 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.535571 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w7hv4_57a1a81e-185f-45c0-b337-947b78db271a/init-textfile/0.log" Sep 09 21:09:15.589963 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.589580 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df9cc_0a7b348e-aec4-4099-9660-e66def008548/kube-rbac-proxy-main/0.log" Sep 09 21:09:15.645757 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.645726 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df9cc_0a7b348e-aec4-4099-9660-e66def008548/kube-rbac-proxy-self/0.log" Sep 09 21:09:15.944889 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:15.944857 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df9cc_0a7b348e-aec4-4099-9660-e66def008548/openshift-state-metrics/0.log" Sep 09 21:09:16.092492 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.092452 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/prometheus/0.log" Sep 09 21:09:16.153866 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.153832 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/config-reloader/0.log" Sep 09 21:09:16.220687 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.220613 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/thanos-sidecar/0.log" Sep 09 21:09:16.276321 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.276293 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/kube-rbac-proxy-web/0.log" Sep 09 21:09:16.320538 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.320507 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/kube-rbac-proxy/0.log" Sep 09 21:09:16.361960 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.361927 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/kube-rbac-proxy-thanos/0.log" Sep 09 21:09:16.410586 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.410554 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e9c3ee39-db57-499f-8e33-ee7b2778106e/init-config-reloader/0.log" Sep 09 21:09:16.456655 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.456620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-b26p7_61899c1a-a222-421a-bf70-c85c602887d5/prometheus-operator/0.log" Sep 09 21:09:16.487491 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.487399 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-b26p7_61899c1a-a222-421a-bf70-c85c602887d5/kube-rbac-proxy/0.log" Sep 09 21:09:16.528469 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.528442 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-hpjxd_d8e6ba84-31bf-491a-b07b-5049b457811d/prometheus-operator-admission-webhook/0.log" Sep 09 21:09:16.565419 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.565389 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85885fb465-7pmvj_d3b415b3-75cf-4e0c-89d6-300ed754f65c/telemeter-client/0.log" Sep 09 21:09:16.600355 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.600324 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85885fb465-7pmvj_d3b415b3-75cf-4e0c-89d6-300ed754f65c/reload/0.log" Sep 09 21:09:16.634347 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.634318 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85885fb465-7pmvj_d3b415b3-75cf-4e0c-89d6-300ed754f65c/kube-rbac-proxy/0.log" Sep 09 21:09:16.671958 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.671910 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/thanos-query/0.log" Sep 09 21:09:16.697454 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.697418 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/kube-rbac-proxy-web/0.log" Sep 09 21:09:16.725851 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.725819 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/kube-rbac-proxy/0.log" Sep 09 21:09:16.755348 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.755320 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/prom-label-proxy/0.log" Sep 09 21:09:16.781176 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.781134 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/kube-rbac-proxy-rules/0.log" Sep 09 21:09:16.806631 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:16.806602 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8dd57b9bc-rlr6m_ce16a656-448c-4447-b1a6-074254c4b4d6/kube-rbac-proxy-metrics/0.log" Sep 09 21:09:17.679135 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.679080 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x"] Sep 09 21:09:17.683156 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.683132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.700149 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.700108 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x"] Sep 09 21:09:17.741695 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.741661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-podres\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.741898 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.741712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-proc\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.741898 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.741740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-lib-modules\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.741898 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.741800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-sys\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.741898 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.741837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmfd\" (UniqueName: \"kubernetes.io/projected/25e48248-170a-4fd0-9634-59978fac35d2-kube-api-access-8rmfd\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843233 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-podres\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-proc\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-lib-modules\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-sys\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843372 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-podres\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843390 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-proc\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843428 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmfd\" (UniqueName: \"kubernetes.io/projected/25e48248-170a-4fd0-9634-59978fac35d2-kube-api-access-8rmfd\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843673 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843448 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-sys\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.843673 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.843497 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25e48248-170a-4fd0-9634-59978fac35d2-lib-modules\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.853997 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.853961 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmfd\" (UniqueName: \"kubernetes.io/projected/25e48248-170a-4fd0-9634-59978fac35d2-kube-api-access-8rmfd\") pod \"perf-node-gather-daemonset-zfs4x\" (UID: \"25e48248-170a-4fd0-9634-59978fac35d2\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:17.997054 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:17.996962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:18.300807 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:18.300781 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x"] Sep 09 21:09:18.303335 ip-10-0-141-196 kubenswrapper[2573]: W0909 21:09:18.303295 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25e48248_170a_4fd0_9634_59978fac35d2.slice/crio-f6f63f3f1d783dd80f81d70fcc5194a82b83d60a1397e843c775e70684b9fbfa WatchSource:0}: Error finding container f6f63f3f1d783dd80f81d70fcc5194a82b83d60a1397e843c775e70684b9fbfa: Status 404 returned error can't find the container with id f6f63f3f1d783dd80f81d70fcc5194a82b83d60a1397e843c775e70684b9fbfa Sep 09 21:09:19.040458 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:19.040424 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d9d87549-9l4s2_780115ae-a50b-4cf7-9825-4de1f789abf0/console/0.log" Sep 09 21:09:19.091731 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:19.091698 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" event={"ID":"25e48248-170a-4fd0-9634-59978fac35d2","Type":"ContainerStarted","Data":"8e640580a39c3a0de51e1d1daef17f6b0f2516a8f00803603f61ded2f93c8da3"} Sep 09 21:09:19.091904 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:19.091759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" event={"ID":"25e48248-170a-4fd0-9634-59978fac35d2","Type":"ContainerStarted","Data":"f6f63f3f1d783dd80f81d70fcc5194a82b83d60a1397e843c775e70684b9fbfa"} Sep 09 21:09:19.091904 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:19.091895 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:19.111322 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:19.111261 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" podStartSLOduration=2.111213709 podStartE2EDuration="2.111213709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:19.108615518 +0000 UTC m=+4445.627495246" watchObservedRunningTime="2026-09-09 21:09:19.111213709 +0000 UTC m=+4445.630093423" Sep 09 21:09:20.909374 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:20.909342 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9vzms_5b2043cc-0dd5-48b3-b0a8-4681c68594dd/dns/0.log" Sep 09 21:09:20.945727 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:20.945686 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9vzms_5b2043cc-0dd5-48b3-b0a8-4681c68594dd/kube-rbac-proxy/0.log" Sep 09 21:09:21.108979 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:21.108953 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klb6w_fb7783cd-ee82-4563-8ad0-35b71502baa3/dns-node-resolver/0.log" Sep 09 21:09:21.791917 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:21.791885 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6tzfm_7d11c2e4-89ae-4d07-89d6-5e44a235076c/node-ca/0.log" Sep 09 21:09:23.762114 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:23.762089 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5fp5p_35286300-d6c0-42e7-83f2-9c9ecec02a87/serve-healthcheck-canary/0.log" Sep 09 21:09:24.353784 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:24.353750 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4z8tv_8bf7caa8-a362-4cf9-82a3-d57f577e21e0/kube-rbac-proxy/0.log" Sep 09 21:09:24.383775 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:24.383747 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4z8tv_8bf7caa8-a362-4cf9-82a3-d57f577e21e0/exporter/0.log" Sep 09 21:09:24.420001 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:24.419976 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4z8tv_8bf7caa8-a362-4cf9-82a3-d57f577e21e0/extractor/0.log" Sep 09 21:09:25.111923 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:25.111896 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-zfs4x" Sep 09 21:09:28.162308 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:28.162279 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7ff45979b5-pswbc_9c9d333b-9f0f-46bc-8dac-0659d9dafd91/manager/0.log" Sep 09 21:09:28.225592 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:28.225521 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8zxk6_3727c8d8-80cc-4929-8dee-29c70e470c33/openshift-lws-operator/0.log" Sep 09 21:09:30.126901 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:30.126866 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8phv8_e28634fd-c46f-47b4-ad53-6e8391e17d43/s3-init/0.log" Sep 09 21:09:30.178399 ip-10-0-141-196 kubenswrapper[2573]: I0909 21:09:30.178365 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8phv8_e28634fd-c46f-47b4-ad53-6e8391e17d43/download-hf-model/0.log"