Sep 11 16:16:49.118925 ip-10-0-135-175 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:16:49.476778 ip-10-0-135-175 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:49.476778 ip-10-0-135-175 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:16:49.476778 ip-10-0-135-175 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:49.476778 ip-10-0-135-175 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:16:49.476778 ip-10-0-135-175 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:49.478385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.478284    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:16:49.482980 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482965    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.482980 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482979    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482984    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482987    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482991    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482994    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482996    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.482999    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483002    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483005    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483008    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483011    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483014    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483016    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483019    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483023    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483027    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483031    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483034    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483036    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.483047 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483040    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483043    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483046    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483048    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483051    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483054    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483057    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483060    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483063    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483065    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483068    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483070    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483073    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483076    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483080    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483084    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483087    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483090    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483093    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.483505 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483095    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483098    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483101    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483103    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483106    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483108    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483111    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483114    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483116    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483119    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483121    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483124    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483126    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483129    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483135    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483139    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483142    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483144    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483147    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483149    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.484017 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483154    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483157    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483160    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483163    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483166    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483169    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483172    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483175    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483177    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483180    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483184    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483187    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483189    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483193    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483195    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483198    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483200    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483203    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483205    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483208    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.484510 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483210    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483213    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483216    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483218    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483221    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483224    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483227    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483236    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483239    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483242    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483245    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483247    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483250    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483252    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483255    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483258    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483260    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483263    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483266    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.483269    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484728    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.485014 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484736    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484740    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484744    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484747    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484750    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484754    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484756    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484759    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484762    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484764    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484767    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484769    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484772    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484775    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484778    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484780    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484783    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484786    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484789    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484792    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.485520 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484795    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484798    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484800    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484804    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484806    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484809    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484811    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484814    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484816    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484819    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484822    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484824    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484826    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484829    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484832    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484834    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484837    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484839    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484842    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.486036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484844    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484847    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484849    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484852    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484856    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484860    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484862    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484865    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484867    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484870    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484873    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484877    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484881    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484884    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484887    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484890    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484893    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484895    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484899    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484902    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.486529 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484904    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484907    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484909    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484912    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484914    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484917    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484921    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484924    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484927    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484933    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484936    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484939    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484942    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484944    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484947    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484949    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484952    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484955    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484957    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484960    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.487041 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484963    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484965    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484968    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484971    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484974    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484978    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484981    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484984    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484986    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484989    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484991    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484994    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.484997    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485000    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485002    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485005    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485008    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485010    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485013    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485960    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.487539 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485967    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485970    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485973    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485976    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485979    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485982    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485985    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485987    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485990    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485993    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485996    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.485999    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486002    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486005    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486008    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486011    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486013    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486016    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486019    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486022    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.488052 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486025    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486028    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486031    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486033    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486036    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486038    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486041    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486044    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486047    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486050    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486053    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486055    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486058    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486060    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486063    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486065    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486068    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486071    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486074    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486077    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.488563 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486080    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486084    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486087    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486090    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486093    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486096    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486098    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486102    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486105    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486108    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486111    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486115    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486118    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486120    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486123    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486125    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486128    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486130    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486133    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.489064 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486135    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486138    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486140    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486143    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486145    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486148    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486150    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486155    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486158    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486161    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486164    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486167    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486169    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486173    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486175    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486178    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486180    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486183    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486185    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486188    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.489646 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486190    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486193    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486195    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486198    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486200    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486203    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486205    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486208    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486210    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486213    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486215    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486218    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486220    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486223    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486225    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486228    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486230    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486233    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486235    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486298    2550 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:16:49.490139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486327    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486333    2550 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486337    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486343    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486346    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486350    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486354    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486357    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486361    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486364    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486367    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486370    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486373    2550 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486376    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486379    2550 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486382    2550 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486385    2550 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486389    2550 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486393    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486396    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486399    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486402    2550 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486406    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:16:49.490629 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486409    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486412    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486415    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486418    2550 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486421    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486424    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486427    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486430    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486434    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486437    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486440    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486443    2550 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486446    2550 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486450    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486455    2550 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486458    2550 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486461    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486464    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486467    2550 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486471    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486474    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486477    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486480    2550 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486483    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486486    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:16:49.491231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486489    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486492    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486495    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486499    2550 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486502    2550 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486505    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486508    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486511    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486514    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486517    2550 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486520    2550 flags.go:64] FLAG: --help="false"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486523    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486526    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486530    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486533    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486537    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486541    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486546    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486549    2550 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486552    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486555    2550 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486558    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486561    2550 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486564    2550 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:16:49.491884 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486567    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486569    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486573    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486575    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486578    2550 flags.go:64] FLAG: --lock-file=""
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486581    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486584    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486587    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486592    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486595    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486598    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486601    2550 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486605    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486608    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486611    2550 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486614    2550 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486618    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486639    2550 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486643    2550 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486646    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486649    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486652    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486655    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486658    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486661    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:16:49.492484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486667    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486674    2550 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486677    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486680    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486683    2550 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486686    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486691    2550 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486694    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486697    2550 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486700    2550 flags.go:64] FLAG: --port="10250"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486703    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486706    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ca7d1be6a88b8ac"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486709    2550 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486712    2550 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486715    2550 flags.go:64] FLAG: --register-node="true"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486718    2550 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486721    2550 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486724    2550 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486727    2550 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486732    2550 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486736    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486739    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486742    2550 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486745    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486748    2550 flags.go:64] FLAG: --runonce="false"
Sep 11 16:16:49.493123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486751    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486754    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486757    2550 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486760    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486762    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486766    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486768    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486772    2550 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486776    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486779    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486781    2550 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486784    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486787    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486790    2550 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486793    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486799    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486801    2550 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486804    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486808    2550 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486811    2550 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486814    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486817    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486819    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486822    2550 flags.go:64] FLAG: --v="2"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486826    2550 flags.go:64] FLAG: --version="false"
Sep 11 16:16:49.493873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486830    2550 flags.go:64] FLAG: --vmodule=""
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486836    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.486839    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486979    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486983    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486987    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486990    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486993    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486996    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.486998    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487001    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487007    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487010    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487013    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487015    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487020    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487022    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487025    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487027    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487030    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.494811 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487033    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487036    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487039    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487042    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487045    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487048    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487050    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487053    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487056    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487060    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487063    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487066    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487069    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487073    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487075    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487078    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487081    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487084    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487087    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.495368 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487089    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487092    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487094    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487097    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487100    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487102    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487105    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487108    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487111    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487114    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487117    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487121    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487125    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487128    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487130    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487133    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487136    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487139    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487141    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487144    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.495865 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487146    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487149    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487152    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487156    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487159    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487162    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487167    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487170    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487173    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487175    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487179    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487181    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487184    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487186    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487189    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487192    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487194    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487197    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487200    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487203    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.496355 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487207    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487209    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487212    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487215    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487217    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487220    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487222    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487225    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487228    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487230    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487233    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487235    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487238    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487241    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487243    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487246    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487248    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487251    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487255    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487257    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.496872 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487260    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487262    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.487265    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.487881    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.495799    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.495814    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.496594    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.496601    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496676    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496682    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496686    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496689    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496693    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496696    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496699    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.497360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496702    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496704    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496707    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496710    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496713    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496716    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496718    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496721    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496724    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496726    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496729    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496732    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496735    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496737    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496744    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496747    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496750    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496752    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496755    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496757    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.497759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496760    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496763    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496766    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496769    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496772    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496775    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496779    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496783    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496786    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496789    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496792    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496795    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496798    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496800    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496803    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496806    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496808    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496811    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496813    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496816    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.498254 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496819    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496822    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496824    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496827    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496829    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496832    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496835    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496838    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496841    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496843    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496846    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496849    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496851    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496854    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496857    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496861    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496864    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496867    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496870    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.498797 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496873    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496876    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496878    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496881    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496885    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496889    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496892    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496894    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496897    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496899    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496902    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496905    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496908    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496911    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496914    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496916    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496919    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496921    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496924    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.499272 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496926    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496929    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496931    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496934    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496937    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496940    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496942    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496945    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496947    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496950    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496952    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496955    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496958    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.496960    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.496965    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:49.499758 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497074    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497079    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497082    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497085    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497088    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497091    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497093    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497096    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497099    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497101    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497104    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497106    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497109    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497112    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497114    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497117    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497119    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497122    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497124    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497127    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:49.500171 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497129    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497132    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497135    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497137    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497140    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497142    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497145    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497148    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497150    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497153    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497156    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497158    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497161    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497164    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497166    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497169    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497172    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497174    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497177    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:49.500684 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497179    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497182    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497185    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497188    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497190    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497193    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497196    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497198    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497201    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497205    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497208    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497211    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497214    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497217    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497220    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497223    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497225    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497228    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497231    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:49.501227 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497234    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497236    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497239    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497241    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497244    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497246    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497249    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497251    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497254    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497257    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497259    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497263    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497266    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497269    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497272    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497275    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497278    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497281    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497283    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:49.501792 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497286    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497290    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497293    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497295    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497298    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497301    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497304    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497307    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497310    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497313    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497315    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497319    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497321    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497325    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497328    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497331    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497334    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497337    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497339    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497342    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:49.502250 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497344    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:49.497347    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.497352    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.497453    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.499704    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.500545    2550 server.go:1019] "Starting client certificate rotation"
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.500667    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.500724    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:49.503039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.502809    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:49.503968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.503949    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:49.517919 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.517900    2550 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:16:49.520722 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.520705    2550 log.go:25] "Validated CRI v1 image API"
Sep 11 16:16:49.521756 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.521741    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:16:49.524708 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.524591    2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b7200d85-7d8d-4247-8269-2c997b6d074c:/dev/nvme0n1p3 d534fffc-f492-48a3-9a63-8079c081dd50:/dev/nvme0n1p4]
Sep 11 16:16:49.524785 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.524707    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:16:49.530169 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.529958    2550 manager.go:216] Machine: {Timestamp:2026-09-11 16:16:49.529146111 +0000 UTC m=+0.303846680 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108434 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222cd8411a0978f9678f8705ad5e07 SystemUUID:ec222cd8-411a-0978-f967-8f8705ad5e07 BootID:ff21fb5f-6a61-4c46-8b7e-12e91c7fd61f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:26:a0:e0:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:26:a0:e0:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:c4:69:e2:9b:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:16:49.530260 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.530177    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:16:49.530473 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.530442    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:16:49.531221 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.531204    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:49.533339 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.533318    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:16:49.533459 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.533342    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:16:49.534706 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.534696    2550 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:16:49.534741 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.534710    2550 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:16:49.534741 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.534717    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:49.535368 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.535358    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:49.536527 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.536518    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:16:49.540768 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540757    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:49.540879 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540870    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:16:49.540907 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540899    2550 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:16:49.540943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540907    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:16:49.540943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540924    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:16:49.540943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540933    2550 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:16:49.540943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.540942    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:16:49.541290 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.541278    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:49.541325 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.541298    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:49.543421 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.543405    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:16:49.546008 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.545993    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:16:49.546079 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.546023    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:16:49.548013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.547999    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:16:49.548013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548016    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548023    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548029    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548038    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548043    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548055    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548061    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548067    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548072    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548081    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548086    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548117    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:16:49.548125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.548122    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:16:49.551220 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.551202    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-175.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:16:49.551746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.551733    2550 server.go:1268] "Started kubelet"
Sep 11 16:16:49.551811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.551786    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:16:49.551969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.551938    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:16:49.552012 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.551988    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:16:49.552076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.552053    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:16:49.552133 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.552058    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:16:49.552644 ip-10-0-135-175 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:16:49.553561 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.553547    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:16:49.556550 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.556538    2550 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:16:49.559205 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.559180    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:16:49.560167 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.560148    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:49.560581 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.560562    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:16:49.560946 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.560747    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.561341 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.560412    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-175.ec2.internal.18d45073a28f80b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-175.ec2.internal,UID:ip-10-0-135-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-175.ec2.internal,},FirstTimestamp:2026-09-11 16:16:49.551704243 +0000 UTC m=+0.326404813,LastTimestamp:2026-09-11 16:16:49.551704243 +0000 UTC m=+0.326404813,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-175.ec2.internal,}"
Sep 11 16:16:49.561425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561385    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:16:49.561425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561389    2550 factory.go:55] Registering systemd factory
Sep 11 16:16:49.561425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561411    2550 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:16:49.561558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561386    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:16:49.561558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561453    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:16:49.561558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561516    2550 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:16:49.561558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561530    2550 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:16:49.561747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561671    2550 factory.go:153] Registering CRI-O factory
Sep 11 16:16:49.561747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561686    2550 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:16:49.561747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561733    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:16:49.561887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561753    2550 factory.go:103] Registering Raw factory
Sep 11 16:16:49.561887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.561767    2550 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:16:49.562115 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.562102    2550 manager.go:349] Starting recovery of all containers
Sep 11 16:16:49.566592 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.566545    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:16:49.566712 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.566686    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:16:49.569753 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.569724    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:16:49.573639 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.573602    2550 manager.go:354] Recovery completed
Sep 11 16:16:49.578790 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.578777    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.581638 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.581595    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.581699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.581653    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.581699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.581665    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.582203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582186    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:16:49.582203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582199    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.582316 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582202    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:16:49.582316 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582274    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:49.582920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582906    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.582980 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582932    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.582980 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.582941    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.583931 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.583875    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-175.ec2.internal.18d45073a4583d2f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-175.ec2.internal,UID:ip-10-0-135-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-175.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-175.ec2.internal,},FirstTimestamp:2026-09-11 16:16:49.581636911 +0000 UTC m=+0.356337484,LastTimestamp:2026-09-11 16:16:49.581636911 +0000 UTC m=+0.356337484,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-175.ec2.internal,}"
Sep 11 16:16:49.584413 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.584402    2550 policy_none.go:49] "None policy: Start"
Sep 11 16:16:49.584449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.584417    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:16:49.584449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.584426    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:16:49.586340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.586330    2550 policy_none.go:47] "Start"
Sep 11 16:16:49.590211 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.590187    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vf2g"
Sep 11 16:16:49.594976 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.594912    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-175.ec2.internal.18d45073a45893fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-175.ec2.internal,UID:ip-10-0-135-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-175.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-175.ec2.internal,},FirstTimestamp:2026-09-11 16:16:49.581659134 +0000 UTC m=+0.356359702,LastTimestamp:2026-09-11 16:16:49.581659134 +0000 UTC m=+0.356359702,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-175.ec2.internal,}"
Sep 11 16:16:49.598905 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.598889    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5vf2g"
Sep 11 16:16:49.605839 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.605771    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-175.ec2.internal.18d45073a458b846  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-175.ec2.internal,UID:ip-10-0-135-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-175.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-175.ec2.internal,},FirstTimestamp:2026-09-11 16:16:49.581668422 +0000 UTC m=+0.356368991,LastTimestamp:2026-09-11 16:16:49.581668422 +0000 UTC m=+0.356368991,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-175.ec2.internal,}"
Sep 11 16:16:49.625232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625209    2550 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.625243    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625253    2550 server.go:87] "Starting device plugin registration server"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625583    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625594    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625924    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625978    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.625984    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.626684    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:16:49.645824 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.626731    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.680460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.680426    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:16:49.680586 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.680489    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:16:49.680586 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.680514    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:16:49.680702 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.680578    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:16:49.683582 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.683562    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:49.726013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.725991    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.727909 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.727835    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.727909 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.727863    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.727909 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.727874    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.727909 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.727897    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.737910 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.737896    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.737955 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.737918    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-175.ec2.internal\": node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.753168 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.753143    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.780800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.780780    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"]
Sep 11 16:16:49.780854 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.780838    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.781668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.781655    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.781750 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.781679    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.781750 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.781688    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.783820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.783808    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.783934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.783920    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.783981 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.783953    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.784474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784458    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.784525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784485    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.784525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784501    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.784525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784506    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.784648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784533    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.784648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.784546    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.786648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.786616    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.786742 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.786657    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:49.787300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.787285    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:49.787355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.787308    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:49.787355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.787317    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:49.802110 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.802091    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-175.ec2.internal\" not found" node="ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.806458 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.806444    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-175.ec2.internal\" not found" node="ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.853788 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.853771    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.863118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.863090    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a31c9521de76f63a6f298dd8cdc84d1f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-175.ec2.internal\" (UID: \"a31c9521de76f63a6f298dd8cdc84d1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.863186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.863123    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.863186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.863139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.954604 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:49.954579    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:49.963939 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.963920    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.964021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.963937    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.964021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.963962    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.964021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.963996    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfb037e9239cf49e2fc99a083a3892a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal\" (UID: \"dfb037e9239cf49e2fc99a083a3892a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.964021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.964005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a31c9521de76f63a6f298dd8cdc84d1f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-175.ec2.internal\" (UID: \"a31c9521de76f63a6f298dd8cdc84d1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:49.964154 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:49.964078    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a31c9521de76f63a6f298dd8cdc84d1f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-175.ec2.internal\" (UID: \"a31c9521de76f63a6f298dd8cdc84d1f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:50.055354 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.055298    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.105743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.105719    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:50.111118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.110834    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:50.156029 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.156007    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.256550 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.256526    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.357036 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.356984    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.457634 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.457604    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.500029 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.500007    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:16:50.500417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.500166    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:16:50.557917 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.557891    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.604623 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.604462    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:11:49 +0000 UTC" deadline="2028-04-18 08:27:11.728598768 +0000 UTC"
Sep 11 16:16:50.604623 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.604618    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14032h10m21.12398476s"
Sep 11 16:16:50.659113 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.659056    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.661283 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.661266    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:50.662425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.662408    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:50.662513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.662438    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:50.662513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.662450    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:50.729180 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:50.729142    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31c9521de76f63a6f298dd8cdc84d1f.slice/crio-92037dd0e8bb91491eb54ea8391d24cc6d2f550f5d04d796173f6f4cfe813caa WatchSource:0}: Error finding container 92037dd0e8bb91491eb54ea8391d24cc6d2f550f5d04d796173f6f4cfe813caa: Status 404 returned error can't find the container with id 92037dd0e8bb91491eb54ea8391d24cc6d2f550f5d04d796173f6f4cfe813caa
Sep 11 16:16:50.729385 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:50.729352    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb037e9239cf49e2fc99a083a3892a6.slice/crio-cf54ea90a87e5096ab6a1e611a9d19ed60b304598c4e1228b6387d84827b6c75 WatchSource:0}: Error finding container cf54ea90a87e5096ab6a1e611a9d19ed60b304598c4e1228b6387d84827b6c75: Status 404 returned error can't find the container with id cf54ea90a87e5096ab6a1e611a9d19ed60b304598c4e1228b6387d84827b6c75
Sep 11 16:16:50.733716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.733679    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:16:50.759827 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:50.759799    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-175.ec2.internal\" not found"
Sep 11 16:16:50.810886 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.810864    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:50.863514 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.863470    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:50.879859 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.879834    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:50.881893 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.881878    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal"
Sep 11 16:16:50.892036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.892018    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:50.934676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.934618    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:50.969316 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:50.969293    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:51.542482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.542449    2550 apiserver.go:52] "Watching apiserver"
Sep 11 16:16:51.560185 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.560156    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:16:51.560420 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.560391    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:51.561455 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.561416    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-bfjtf","openshift-image-registry/node-ca-ccx2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal","openshift-multus/network-metrics-daemon-wj4ht","kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal","openshift-multus/multus-additional-cni-plugins-rgg4t","openshift-multus/multus-dxhpb","openshift-network-diagnostics/network-check-target-8pwdf","openshift-network-operator/iptables-alerter-nvrdp","openshift-ovn-kubernetes/ovnkube-node-6rgj5","kube-system/global-pull-secret-syncer-6wc5s","kube-system/konnectivity-agent-6qbgc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz","openshift-cluster-node-tuning-operator/tuned-rxld8"]
Sep 11 16:16:51.564413 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.564381    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.566794 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.566771    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.566897 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.566862    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:16:51.567726 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.567702    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:16:51.567822 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.567743    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9r49b\""
Sep 11 16:16:51.567889 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.567823    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:16:51.569087 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.569070    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.569470 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.569450    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.569564 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.569538    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.571377 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.571357    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.571475 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.571421    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:16:51.571880 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.571861    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kjtcq\""
Sep 11 16:16:51.573617 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.573599    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.573725 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.573658    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.573992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.573976    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:51.574076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.574053    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:16:51.576452 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.576433    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.576820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.576804    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:51.579122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.579096    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.579122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.579104    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:16:51.582634 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.580787    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5h2pr\""
Sep 11 16:16:51.582634 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.581061    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:16:51.582812 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.582612    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q6jbx\""
Sep 11 16:16:51.582868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.582818    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.583980 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.583962    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.584097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.583989    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:16:51.584097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.583965    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.585958 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.585937    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.586047 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.586011    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.586420 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.586399    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:16:51.586516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.586458    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sqgqx\""
Sep 11 16:16:51.586584 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.586549    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.588525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.588506    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.589110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.589085    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:16:51.589227 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.589109    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmllf\""
Sep 11 16:16:51.589339 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.589321    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.589339 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.589349    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.591231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.591132    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.592808 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.592786    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.592915 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.592805    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:16:51.592915 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.592859    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hxbmp\""
Sep 11 16:16:51.592915 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.592904    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:16:51.593443 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.593427    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.594477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.594459    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.595180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.594683    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:16:51.595180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.594729    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:16:51.595180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.594751    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:16:51.595180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.594768    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:16:51.595180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.595071    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qh2x\""
Sep 11 16:16:51.596347 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.596316    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:16:51.597781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.597762    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:16:51.597877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.597862    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gcbzg\""
Sep 11 16:16:51.610537 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.610510    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vm267"
Sep 11 16:16:51.617104 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.617076    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vm267"
Sep 11 16:16:51.663308 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.663287    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:16:51.671838 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-cni-binary-copy\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.671950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671839    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysconfig\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.671950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-etc-selinux\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.671950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671884    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-slash\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.671950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671908    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5c2285-c033-4362-abc2-0674294c8a2a-ovn-node-metrics-cert\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-dbus\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.671987    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672011    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672061    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-sys-fs\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672108    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.672162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cnibin\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672166    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stj56\" (UniqueName: \"kubernetes.io/projected/3e5c2285-c033-4362-abc2-0674294c8a2a-kube-api-access-stj56\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672193    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-system-cni-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-netns\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672258    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672286    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2tj\" (UniqueName: \"kubernetes.io/projected/4c415c27-e7dd-4eae-a2ea-234ec2c85631-kube-api-access-dv2tj\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672319    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9e08126-0dfc-43dc-a899-0597def90b74-konnectivity-ca\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-etc-kubernetes\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6msm\" (UniqueName: \"kubernetes.io/projected/bebfd61b-c763-412f-8bb5-8560749f62b1-kube-api-access-p6msm\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672394    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-systemd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672419    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-node-log\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672442    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-log-socket\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672465    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-bin\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672505    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-env-overrides\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-socket-dir-parent\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672555    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-tmp\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672577    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f68322-bf74-4f70-bf0b-78a549ddce4b-host-slash\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672617    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672667    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672690    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-hostroot\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672730    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672771    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5h97\" (UniqueName: \"kubernetes.io/projected/e2f68322-bf74-4f70-bf0b-78a549ddce4b-kube-api-access-b5h97\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-var-lib-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672812    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f85z\" (UniqueName: \"kubernetes.io/projected/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-kube-api-access-9f85z\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672833    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-modprobe-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672847    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-kubernetes\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7379ca9e-d28c-4754-926d-f13edc395a9c-host\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672874    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.672921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672890    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-tmp-dir\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672913    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-k8s-cni-cncf-io\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672940    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92vl\" (UniqueName: \"kubernetes.io/projected/e6234a20-e5b0-4098-9721-e0e098e2528b-kube-api-access-d92vl\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672963    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-etc-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.672986    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673001    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-cnibin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673023    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-sys\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673047    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673065    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2f68322-bf74-4f70-bf0b-78a549ddce4b-iptables-alerter-script\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-registration-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673103    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6j4d\" (UniqueName: \"kubernetes.io/projected/598efbc0-8955-4220-8a1b-28a652c3e20f-kube-api-access-d6j4d\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673129    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-netd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673155    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-multus\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673179    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673208    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-hosts-file\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673223    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9e08126-0dfc-43dc-a899-0597def90b74-agent-certs\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.673625 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673247    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-system-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7379ca9e-d28c-4754-926d-f13edc395a9c-serviceca\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673286    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-script-lib\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6g6f\" (UniqueName: \"kubernetes.io/projected/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-kube-api-access-s6g6f\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-netns\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673354    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-kubelet\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673728    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-conf\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673819    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-run\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673851    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-host\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.673881    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-socket-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674020    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-device-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674055    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-ovn\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-bin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674110    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-multus-certs\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674141    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-systemd\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674171    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-var-lib-kubelet\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwp4b\" (UniqueName: \"kubernetes.io/projected/7379ca9e-d28c-4754-926d-f13edc395a9c-kube-api-access-pwp4b\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.674425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674227    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-kubelet\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674256    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-systemd-units\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674305    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-kubelet-config\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674337    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-lib-modules\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-conf-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674392    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-daemon-config\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674423    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-tuned\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674478    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.675226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.674537    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-config\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.683880 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.683863    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:51.685516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.685456    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal" event={"ID":"a31c9521de76f63a6f298dd8cdc84d1f","Type":"ContainerStarted","Data":"92037dd0e8bb91491eb54ea8391d24cc6d2f550f5d04d796173f6f4cfe813caa"}
Sep 11 16:16:51.686896 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.686869    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal" event={"ID":"dfb037e9239cf49e2fc99a083a3892a6","Type":"ContainerStarted","Data":"cf54ea90a87e5096ab6a1e611a9d19ed60b304598c4e1228b6387d84827b6c75"}
Sep 11 16:16:51.775520 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775490    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-tuned\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775529    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775596    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775622    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-config\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775666    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-cni-binary-copy\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775689    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysconfig\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775695    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.775720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-etc-selinux\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-slash\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775758    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5c2285-c033-4362-abc2-0674294c8a2a-ovn-node-metrics-cert\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775781    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-dbus\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775857    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775886    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-sys-fs\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775910    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775935    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cnibin\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.775960    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stj56\" (UniqueName: \"kubernetes.io/projected/3e5c2285-c033-4362-abc2-0674294c8a2a-kube-api-access-stj56\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-system-cni-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776031    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-netns\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776076    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776104    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2tj\" (UniqueName: \"kubernetes.io/projected/4c415c27-e7dd-4eae-a2ea-234ec2c85631-kube-api-access-dv2tj\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776144    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9e08126-0dfc-43dc-a899-0597def90b74-konnectivity-ca\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-etc-kubernetes\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776190    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-cni-binary-copy\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776214    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6msm\" (UniqueName: \"kubernetes.io/projected/bebfd61b-c763-412f-8bb5-8560749f62b1-kube-api-access-p6msm\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776243    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-systemd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776271    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-node-log\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776276    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776291    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-run-netns\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776309    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776332    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-node-log\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysconfig\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776400    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cnibin\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776414    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-etc-selinux\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776436    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776488    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-log-socket\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776525    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-bin\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.776764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-env-overrides\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.776563    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776580    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-socket-dir-parent\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776608    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-tmp\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776616    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c415c27-e7dd-4eae-a2ea-234ec2c85631-system-cni-dir\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f68322-bf74-4f70-bf0b-78a549ddce4b-host-slash\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.776693    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.276638801 +0000 UTC m=+3.051339377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776728    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-etc-kubernetes\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776734    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f68322-bf74-4f70-bf0b-78a549ddce4b-host-slash\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776703    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-slash\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-systemd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776785    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776813    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776830    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-log-socket\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776840    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-hostroot\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776851    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-dbus\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776857    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-config\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776866    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.777715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776919    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-hostroot\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776927    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-sys-fs\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776921    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776948    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5h97\" (UniqueName: \"kubernetes.io/projected/e2f68322-bf74-4f70-bf0b-78a549ddce4b-kube-api-access-b5h97\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776973    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-var-lib-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.776997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f85z\" (UniqueName: \"kubernetes.io/projected/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-kube-api-access-9f85z\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777019    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777061    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-var-lib-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777088    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-socket-dir-parent\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777094    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-modprobe-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777022    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-modprobe-d\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777226    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-kubernetes\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777250    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7379ca9e-d28c-4754-926d-f13edc395a9c-host\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777283    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-env-overrides\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777301    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-tmp-dir\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777289    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777327    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-k8s-cni-cncf-io\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.778645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777225    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9e08126-0dfc-43dc-a899-0597def90b74-konnectivity-ca\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777354    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92vl\" (UniqueName: \"kubernetes.io/projected/e6234a20-e5b0-4098-9721-e0e098e2528b-kube-api-access-d92vl\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777380    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-etc-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777403    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777427    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-cnibin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777436    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7379ca9e-d28c-4754-926d-f13edc395a9c-host\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777450    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-sys\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777489    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777503    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-sys\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2f68322-bf74-4f70-bf0b-78a549ddce4b-iptables-alerter-script\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777544    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-k8s-cni-cncf-io\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777548    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-registration-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-bin\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777578    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6j4d\" (UniqueName: \"kubernetes.io/projected/598efbc0-8955-4220-8a1b-28a652c3e20f-kube-api-access-d6j4d\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777590    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-tmp-dir\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777635    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-netd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777671    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-multus\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.779922 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.777731    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777734    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777790    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-hosts-file\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-kubernetes\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777835    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9e08126-0dfc-43dc-a899-0597def90b74-agent-certs\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777863    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-system-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777890    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7379ca9e-d28c-4754-926d-f13edc395a9c-serviceca\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777893    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-cnibin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777893    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-script-lib\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777938    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-etc-openvswitch\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777948    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6g6f\" (UniqueName: \"kubernetes.io/projected/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-kube-api-access-s6g6f\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.777977    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-netns\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778002    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-hosts-file\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778002    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-kubelet\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778044    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-conf\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778066    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-kubelet\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778071    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-run\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.780781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-host\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778113    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-run\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778163    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-host\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778262    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-registration-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778461    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c415c27-e7dd-4eae-a2ea-234ec2c85631-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778589    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7379ca9e-d28c-4754-926d-f13edc395a9c-serviceca\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.778633    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.27861337 +0000 UTC m=+3.053313935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778680    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e5c2285-c033-4362-abc2-0674294c8a2a-ovnkube-script-lib\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778745    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-cni-netd\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778785    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-system-cni-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778794    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-multus\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778825    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-sysctl-conf\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778825    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-socket-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778844    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-netns\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-device-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778927    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-ovn\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.781641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778959    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-bin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778966    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-run-ovn\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778969    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-device-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778984    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-multus-certs\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.778928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/598efbc0-8955-4220-8a1b-28a652c3e20f-socket-dir\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779009    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-systemd\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-var-lib-kubelet\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779036    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-var-lib-cni-bin\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779060    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwp4b\" (UniqueName: \"kubernetes.io/projected/7379ca9e-d28c-4754-926d-f13edc395a9c-kube-api-access-pwp4b\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779044    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-host-run-multus-certs\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779086    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-kubelet\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779090    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-systemd\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779039    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e2f68322-bf74-4f70-bf0b-78a549ddce4b-iptables-alerter-script\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779115    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-systemd-units\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779119    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-var-lib-kubelet\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779141    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-kubelet-config\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779155    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-host-kubelet\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-lib-modules\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779220    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a41d8fe-c043-4cf0-b634-afbb8447d967-kubelet-config\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779223    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-conf-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779261    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-daemon-config\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779282    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bebfd61b-c763-412f-8bb5-8560749f62b1-lib-modules\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779176    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e5c2285-c033-4362-abc2-0674294c8a2a-systemd-units\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779270    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-conf-dir\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.779767    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e6234a20-e5b0-4098-9721-e0e098e2528b-multus-daemon-config\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.781203    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-etc-tuned\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.781778    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebfd61b-c763-412f-8bb5-8560749f62b1-tmp\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.781927    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e5c2285-c033-4362-abc2-0674294c8a2a-ovn-node-metrics-cert\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.782986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.782059    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9e08126-0dfc-43dc-a899-0597def90b74-agent-certs\") pod \"konnectivity-agent-6qbgc\" (UID: \"a9e08126-0dfc-43dc-a899-0597def90b74\") " pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:51.786904 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.786881    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:51.786988 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.786908    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:51.786988 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.786923    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:51.786988 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:51.786985    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.286963917 +0000 UTC m=+3.061664500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:51.787185 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.787113    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f85z\" (UniqueName: \"kubernetes.io/projected/29a5160d-5b16-4a1c-aa25-d3e03ef9a013-kube-api-access-9f85z\") pod \"node-resolver-bfjtf\" (UID: \"29a5160d-5b16-4a1c-aa25-d3e03ef9a013\") " pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.787991 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.787920    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6msm\" (UniqueName: \"kubernetes.io/projected/bebfd61b-c763-412f-8bb5-8560749f62b1-kube-api-access-p6msm\") pod \"tuned-rxld8\" (UID: \"bebfd61b-c763-412f-8bb5-8560749f62b1\") " pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.788604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.788216    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5h97\" (UniqueName: \"kubernetes.io/projected/e2f68322-bf74-4f70-bf0b-78a549ddce4b-kube-api-access-b5h97\") pod \"iptables-alerter-nvrdp\" (UID: \"e2f68322-bf74-4f70-bf0b-78a549ddce4b\") " pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.788604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.788252    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6g6f\" (UniqueName: \"kubernetes.io/projected/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-kube-api-access-s6g6f\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:51.788604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.788560    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stj56\" (UniqueName: \"kubernetes.io/projected/3e5c2285-c033-4362-abc2-0674294c8a2a-kube-api-access-stj56\") pod \"ovnkube-node-6rgj5\" (UID: \"3e5c2285-c033-4362-abc2-0674294c8a2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.790033 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.789988    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6j4d\" (UniqueName: \"kubernetes.io/projected/598efbc0-8955-4220-8a1b-28a652c3e20f-kube-api-access-d6j4d\") pod \"aws-ebs-csi-driver-node-clhlz\" (UID: \"598efbc0-8955-4220-8a1b-28a652c3e20f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.790430 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.790405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwp4b\" (UniqueName: \"kubernetes.io/projected/7379ca9e-d28c-4754-926d-f13edc395a9c-kube-api-access-pwp4b\") pod \"node-ca-ccx2w\" (UID: \"7379ca9e-d28c-4754-926d-f13edc395a9c\") " pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.790716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.790680    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92vl\" (UniqueName: \"kubernetes.io/projected/e6234a20-e5b0-4098-9721-e0e098e2528b-kube-api-access-d92vl\") pod \"multus-dxhpb\" (UID: \"e6234a20-e5b0-4098-9721-e0e098e2528b\") " pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.791032 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.791012    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2tj\" (UniqueName: \"kubernetes.io/projected/4c415c27-e7dd-4eae-a2ea-234ec2c85631-kube-api-access-dv2tj\") pod \"multus-additional-cni-plugins-rgg4t\" (UID: \"4c415c27-e7dd-4eae-a2ea-234ec2c85631\") " pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.876855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.876777    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dxhpb"
Sep 11 16:16:51.885322 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.885297    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxld8"
Sep 11 16:16:51.896014 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.895992    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgg4t"
Sep 11 16:16:51.901994 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.901969    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvrdp"
Sep 11 16:16:51.909879 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.909861    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz"
Sep 11 16:16:51.918715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.918691    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccx2w"
Sep 11 16:16:51.927493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.927474    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:16:51.935740 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.935724    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfjtf"
Sep 11 16:16:51.942439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:51.942415    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:16:52.283289 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.283257    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:52.283493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.283300    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:52.283493 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.283424    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.283493 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.283441    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.283493 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.283475    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:16:53.283461512 +0000 UTC m=+4.058162071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.283750 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.283532    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:53.283511852 +0000 UTC m=+4.058212413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.384199 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.384167    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:52.384372 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.384349    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:52.384419 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.384373    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:52.384419 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.384384    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:52.384506 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:52.384440    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:53.38442633 +0000 UTC m=+4.159126893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:52.620840 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.620759    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:51 +0000 UTC" deadline="2028-05-31 08:43:10.998931092 +0000 UTC"
Sep 11 16:16:52.620840 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.620791    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h26m18.378143146s"
Sep 11 16:16:52.683598 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.683048    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6234a20_e5b0_4098_9721_e0e098e2528b.slice/crio-c7043778420140c8097c0d2cd949577ea92d0f875f19e04012e7da1d1f0fde8f WatchSource:0}: Error finding container c7043778420140c8097c0d2cd949577ea92d0f875f19e04012e7da1d1f0fde8f: Status 404 returned error can't find the container with id c7043778420140c8097c0d2cd949577ea92d0f875f19e04012e7da1d1f0fde8f
Sep 11 16:16:52.686213 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.686065    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebfd61b_c763_412f_8bb5_8560749f62b1.slice/crio-10bc60192a365c44702b02fd2f5495172f5fe7d921126e782f0103ab65c16fe6 WatchSource:0}: Error finding container 10bc60192a365c44702b02fd2f5495172f5fe7d921126e782f0103ab65c16fe6: Status 404 returned error can't find the container with id 10bc60192a365c44702b02fd2f5495172f5fe7d921126e782f0103ab65c16fe6
Sep 11 16:16:52.690408 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:52.690340    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxhpb" event={"ID":"e6234a20-e5b0-4098-9721-e0e098e2528b","Type":"ContainerStarted","Data":"c7043778420140c8097c0d2cd949577ea92d0f875f19e04012e7da1d1f0fde8f"}
Sep 11 16:16:52.691557 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.691529    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5c2285_c033_4362_abc2_0674294c8a2a.slice/crio-c4a150fc63a9b48776c822ea7577dfeb20d26be7796f9febfdc43e67dd5b9c68 WatchSource:0}: Error finding container c4a150fc63a9b48776c822ea7577dfeb20d26be7796f9febfdc43e67dd5b9c68: Status 404 returned error can't find the container with id c4a150fc63a9b48776c822ea7577dfeb20d26be7796f9febfdc43e67dd5b9c68
Sep 11 16:16:52.692598 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.692572    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7379ca9e_d28c_4754_926d_f13edc395a9c.slice/crio-e55c8bdb7b3e45b25cda0aa94da7a98b3095fe13b5e9d6399b20c79f55fb3bf6 WatchSource:0}: Error finding container e55c8bdb7b3e45b25cda0aa94da7a98b3095fe13b5e9d6399b20c79f55fb3bf6: Status 404 returned error can't find the container with id e55c8bdb7b3e45b25cda0aa94da7a98b3095fe13b5e9d6399b20c79f55fb3bf6
Sep 11 16:16:52.694021 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.693505    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598efbc0_8955_4220_8a1b_28a652c3e20f.slice/crio-fbf993c364dc012e4eea5e89feaf404393720db40d952b24c02d8eee56c97964 WatchSource:0}: Error finding container fbf993c364dc012e4eea5e89feaf404393720db40d952b24c02d8eee56c97964: Status 404 returned error can't find the container with id fbf993c364dc012e4eea5e89feaf404393720db40d952b24c02d8eee56c97964
Sep 11 16:16:52.701956 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.701233    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f68322_bf74_4f70_bf0b_78a549ddce4b.slice/crio-dc1458fe4fcbeff540b4d6a7b7980d43f2fb6d22eba1e73adb99e75c4ed32b64 WatchSource:0}: Error finding container dc1458fe4fcbeff540b4d6a7b7980d43f2fb6d22eba1e73adb99e75c4ed32b64: Status 404 returned error can't find the container with id dc1458fe4fcbeff540b4d6a7b7980d43f2fb6d22eba1e73adb99e75c4ed32b64
Sep 11 16:16:52.701956 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.701722    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c415c27_e7dd_4eae_a2ea_234ec2c85631.slice/crio-b1e82ae2f2141af558e61f0491e4bef64791e59b3c719ee1e652fdbb31bd12af WatchSource:0}: Error finding container b1e82ae2f2141af558e61f0491e4bef64791e59b3c719ee1e652fdbb31bd12af: Status 404 returned error can't find the container with id b1e82ae2f2141af558e61f0491e4bef64791e59b3c719ee1e652fdbb31bd12af
Sep 11 16:16:52.702993 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:16:52.702967    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e08126_0dfc_43dc_a899_0597def90b74.slice/crio-0fbaafe4d4d79b6decf2cfa4d2f5f1c026aaaf3d8c6603b2f5ea56fe7b3e53da WatchSource:0}: Error finding container 0fbaafe4d4d79b6decf2cfa4d2f5f1c026aaaf3d8c6603b2f5ea56fe7b3e53da: Status 404 returned error can't find the container with id 0fbaafe4d4d79b6decf2cfa4d2f5f1c026aaaf3d8c6603b2f5ea56fe7b3e53da
Sep 11 16:16:53.293163 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.293121    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:53.293358 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.293179    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:53.293429 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.293412    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:53.293496 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.293481    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:16:55.293460475 +0000 UTC m=+6.068161049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:53.293855 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.293831    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:53.293960 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.293899    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:55.293880074 +0000 UTC m=+6.068580636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:53.393533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.393487    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:53.393758 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.393732    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:53.393842 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.393768    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:53.393842 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.393782    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:53.393942 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.393839    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:55.393819877 +0000 UTC m=+6.168520456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:53.622054 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.621933    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:51 +0000 UTC" deadline="2028-04-27 23:49:47.6166421 +0000 UTC"
Sep 11 16:16:53.622054 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.621995    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14263h32m53.994650804s"
Sep 11 16:16:53.683760 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.683728    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:53.683942 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.683864    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:16:53.684373 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.684352    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:53.684496 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.684470    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:16:53.684558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.684535    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:53.684624 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:53.684603    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:16:53.709196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.709132    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal" event={"ID":"dfb037e9239cf49e2fc99a083a3892a6","Type":"ContainerDied","Data":"c54a95b3d382a0f44ee40302315445d5b53247af55a3cffe11e3f30660dc80cd"}
Sep 11 16:16:53.709404 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.709308    2550 generic.go:356] "Generic (PLEG): container finished" podID="dfb037e9239cf49e2fc99a083a3892a6" containerID="c54a95b3d382a0f44ee40302315445d5b53247af55a3cffe11e3f30660dc80cd" exitCode=0
Sep 11 16:16:53.725952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.725880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qbgc" event={"ID":"a9e08126-0dfc-43dc-a899-0597def90b74","Type":"ContainerStarted","Data":"0fbaafe4d4d79b6decf2cfa4d2f5f1c026aaaf3d8c6603b2f5ea56fe7b3e53da"}
Sep 11 16:16:53.733758 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.733673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvrdp" event={"ID":"e2f68322-bf74-4f70-bf0b-78a549ddce4b","Type":"ContainerStarted","Data":"dc1458fe4fcbeff540b4d6a7b7980d43f2fb6d22eba1e73adb99e75c4ed32b64"}
Sep 11 16:16:53.753270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.753208    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccx2w" event={"ID":"7379ca9e-d28c-4754-926d-f13edc395a9c","Type":"ContainerStarted","Data":"e55c8bdb7b3e45b25cda0aa94da7a98b3095fe13b5e9d6399b20c79f55fb3bf6"}
Sep 11 16:16:53.770356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.770240    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz" event={"ID":"598efbc0-8955-4220-8a1b-28a652c3e20f","Type":"ContainerStarted","Data":"fbf993c364dc012e4eea5e89feaf404393720db40d952b24c02d8eee56c97964"}
Sep 11 16:16:53.791100 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.791070    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal" event={"ID":"a31c9521de76f63a6f298dd8cdc84d1f","Type":"ContainerStarted","Data":"0525a952a3dc9f337cfd6a9b032e01fffe05c07f313d04be219223935e553475"}
Sep 11 16:16:53.801040 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.800802    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerStarted","Data":"b1e82ae2f2141af558e61f0491e4bef64791e59b3c719ee1e652fdbb31bd12af"}
Sep 11 16:16:53.811409 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.811356    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfjtf" event={"ID":"29a5160d-5b16-4a1c-aa25-d3e03ef9a013","Type":"ContainerStarted","Data":"ff28e1d2d48410e8259a1abee8abe11c6110b7705fbbc62f82d3e87cc2fa8272"}
Sep 11 16:16:53.833608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.833574    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"c4a150fc63a9b48776c822ea7577dfeb20d26be7796f9febfdc43e67dd5b9c68"}
Sep 11 16:16:53.844616 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:53.844582    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxld8" event={"ID":"bebfd61b-c763-412f-8bb5-8560749f62b1","Type":"ContainerStarted","Data":"10bc60192a365c44702b02fd2f5495172f5fe7d921126e782f0103ab65c16fe6"}
Sep 11 16:16:54.875860 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:54.875645    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal" event={"ID":"dfb037e9239cf49e2fc99a083a3892a6","Type":"ContainerStarted","Data":"f3a9c0de8404a1f75d22f242da5740235b59b819d9c960d38997932f7d8ee584"}
Sep 11 16:16:54.891028 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:54.890940    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-175.ec2.internal" podStartSLOduration=3.890919073 podStartE2EDuration="3.890919073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:53.807788926 +0000 UTC m=+4.582489511" watchObservedRunningTime="2026-09-11 16:16:54.890919073 +0000 UTC m=+5.665619652"
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.309220    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.309270    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.309406    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.309471    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:16:59.30944645 +0000 UTC m=+10.084147008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.309877    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:55.309997 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.309927    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:59.30991262 +0000 UTC m=+10.084613176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:55.410214 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.409641    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:55.410214 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.409796    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:55.410214 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.409815    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:55.410214 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.409827    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:55.410214 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.409892    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:16:59.409873204 +0000 UTC m=+10.184573775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:55.681824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.681734    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:55.681985 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.681854    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:16:55.682248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.682225    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:55.682350 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.682320    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:16:55.682500 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:55.682484    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:55.682634 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:55.682601    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:16:57.681781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:57.681750    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:57.682242 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:57.681879    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:16:57.682242 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:57.681914    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:57.682242 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:57.681998    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:16:57.682242 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:57.682076    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:57.682242 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:57.682202    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.343295    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.343343    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.343497    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.343555    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.343537767 +0000 UTC m=+18.118238325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.343611    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:59.343716 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.343676    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.343661552 +0000 UTC m=+18.118362123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:59.444362 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.444326    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:59.444547 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.444471    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:59.444547 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.444486    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:59.444547 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.444496    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:59.444547 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.444537    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.444525326 +0000 UTC m=+18.219225882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:59.681589 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.681518    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:16:59.681768 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.681611    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:16:59.681768 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.681637    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:16:59.681768 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.681751    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:16:59.681917 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:16:59.681807    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:16:59.681917 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:16:59.681878    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:01.681646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:01.681593    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:01.681646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:01.681643    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:01.682158 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:01.681594    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:01.682158 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:01.681775    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:01.682158 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:01.681876    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:01.682158 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:01.681940    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:03.681642 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:03.681600    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:03.681642 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:03.681640    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:03.682160 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:03.681600    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:03.682160 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:03.681739    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:03.682160 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:03.681836    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:03.682160 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:03.681921    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:05.681284 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:05.681252    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:05.681749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:05.681252    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:05.681749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:05.681263    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:05.681749 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:05.681393    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:05.681749 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:05.681499    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:05.681749 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:05.681604    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:07.393791 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.393761    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:07.393791 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.393795    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:07.394290 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.393906    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.394290 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.393924    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.394290 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.393983    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.393964976 +0000 UTC m=+34.168665545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.394290 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.393998    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.393992316 +0000 UTC m=+34.168692872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.494306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.494269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:07.494463 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.494438    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:07.494518 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.494471    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:07.494518 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.494481    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.494603 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.494535    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.494518542 +0000 UTC m=+34.269219101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.681338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.681258    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:07.681499 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.681270    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:07.681499 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.681386    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:07.681499 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.681479    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:07.681661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:07.681269    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:07.681661 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:07.681596    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:09.681927 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:09.681894    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:09.682360 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:09.681994    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:09.682360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:09.682043    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:09.682360 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:09.682110    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:09.682360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:09.682181    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:09.682360 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:09.682289    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:10.910008 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.909778    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qbgc" event={"ID":"a9e08126-0dfc-43dc-a899-0597def90b74","Type":"ContainerStarted","Data":"704fc680f42f8ffee07bce118e905325f957fe0c567ced53c670dc5396e351b3"}
Sep 11 16:17:10.911565 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.911508    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccx2w" event={"ID":"7379ca9e-d28c-4754-926d-f13edc395a9c","Type":"ContainerStarted","Data":"6580f10bd8ce16ebfc28d67ed2cc6eb9c0a843fd2394819258ab3fb73d38a10a"}
Sep 11 16:17:10.913216 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.913194    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz" event={"ID":"598efbc0-8955-4220-8a1b-28a652c3e20f","Type":"ContainerStarted","Data":"d220a3018eab4487bcce919b24f81cab3f3eabbe679acea6864fedad8b94a91a"}
Sep 11 16:17:10.914702 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.914679    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="66f27eb53390767f756c53aef3dd6b8db840614b1e114b80c637b7ca578fb3bc" exitCode=0
Sep 11 16:17:10.914786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.914749    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"66f27eb53390767f756c53aef3dd6b8db840614b1e114b80c637b7ca578fb3bc"}
Sep 11 16:17:10.916300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.916284    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfjtf" event={"ID":"29a5160d-5b16-4a1c-aa25-d3e03ef9a013","Type":"ContainerStarted","Data":"37d91b766008ef0a3909f4fcfe1686d599ba36deec7b043e3115a6d90a393a08"}
Sep 11 16:17:10.918673 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.918652    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"3687c27e24d1c269e56f3939101f2971c74e83eed158eb8fc8a261b4fff6b903"}
Sep 11 16:17:10.918766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.918708    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"c808ac5a7dc05caab717d14c336acdfe21023a85081e62c786b5036edb80801f"}
Sep 11 16:17:10.918766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.918745    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"a93555163171b5f8b4ed404c5e6f37de268c9f4ef4b8015ad507ed9794f4cdac"}
Sep 11 16:17:10.920111 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.920085    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxld8" event={"ID":"bebfd61b-c763-412f-8bb5-8560749f62b1","Type":"ContainerStarted","Data":"2db984d7eb86d3bd73f7608d4642a3ebebd76cb42617af59749ab9925c476a7f"}
Sep 11 16:17:10.921488 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.921468    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dxhpb" event={"ID":"e6234a20-e5b0-4098-9721-e0e098e2528b","Type":"ContainerStarted","Data":"b2bd9779a06f6663c7fc5525eb24d928e2e8ce2ef7de4b98ce21217fae603408"}
Sep 11 16:17:10.926025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.925984    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6qbgc" podStartSLOduration=7.145286995 podStartE2EDuration="20.925972904s" totalImagesPullingTime="13.780685909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:10.925535273 +0000 UTC m=+21.700235852" watchObservedRunningTime="2026-09-11 16:17:10.925972904 +0000 UTC m=+21.700673485"
Sep 11 16:17:10.926286 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.926259    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-175.ec2.internal" podStartSLOduration=19.926252301 podStartE2EDuration="19.926252301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:54.891482964 +0000 UTC m=+5.666183543" watchObservedRunningTime="2026-09-11 16:17:10.926252301 +0000 UTC m=+21.700952879"
Sep 11 16:17:10.971129 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.971088    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dxhpb" podStartSLOduration=2.916471629 podStartE2EDuration="20.971072684s" totalImagesPullingTime="18.054601055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:10.97086667 +0000 UTC m=+21.745567247" watchObservedRunningTime="2026-09-11 16:17:10.971072684 +0000 UTC m=+21.745773261"
Sep 11 16:17:10.989200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:10.989158    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rxld8" podStartSLOduration=3.138022652 podStartE2EDuration="20.989142542s" totalImagesPullingTime="17.85111989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:10.988462646 +0000 UTC m=+21.763163226" watchObservedRunningTime="2026-09-11 16:17:10.989142542 +0000 UTC m=+21.763843120"
Sep 11 16:17:11.005931 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.005893    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ccx2w" podStartSLOduration=3.161036296 podStartE2EDuration="21.005880884s" totalImagesPullingTime="17.844844588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:11.005866628 +0000 UTC m=+21.780567206" watchObservedRunningTime="2026-09-11 16:17:11.005880884 +0000 UTC m=+21.780581462"
Sep 11 16:17:11.681024 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.680809    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:11.681204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.680809    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:11.681204 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:11.681130    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:11.681322 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:11.681228    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:11.681322 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.680809    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:11.681426 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:11.681327    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:11.928265 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.928229    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"1d1d93b2e7da99790afaa4f54320bc633ce92003267cc60cd6e673192381106e"}
Sep 11 16:17:11.928753 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.928324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"fca4b4482e844509149440c60ee5927af9bbc3d9161d6862f475a84b5f80afbf"}
Sep 11 16:17:11.928753 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.928376    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"491e0ff97b0d5e09e4fea0e109cf5cb78131ca5070d8485bbfb98b307720f54f"}
Sep 11 16:17:11.930417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.930392    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvrdp" event={"ID":"e2f68322-bf74-4f70-bf0b-78a549ddce4b","Type":"ContainerStarted","Data":"523b71f255410eea7dcc0c9f3b00319f528b4178da66d71f7bdc5bb74046fafd"}
Sep 11 16:17:11.947378 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.947332    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bfjtf" podStartSLOduration=4.108956953 podStartE2EDuration="21.947315843s" totalImagesPullingTime="17.83835889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:11.035427538 +0000 UTC m=+21.810128116" watchObservedRunningTime="2026-09-11 16:17:11.947315843 +0000 UTC m=+22.722016422"
Sep 11 16:17:11.947874 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:11.947844    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nvrdp" podStartSLOduration=4.113059676 podStartE2EDuration="21.947832825s" totalImagesPullingTime="17.834773149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:11.946803488 +0000 UTC m=+22.721504080" watchObservedRunningTime="2026-09-11 16:17:11.947832825 +0000 UTC m=+22.722533404"
Sep 11 16:17:12.091302 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:12.091277    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:17:12.638052 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:12.637950    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:17:12.091296881Z","UUID":"4f140b04-827f-499f-8254-ae3d25cca7c4","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:17:12.639531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:12.639502    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:17:12.639531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:12.639535    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:17:12.935049 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:12.934970    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz" event={"ID":"598efbc0-8955-4220-8a1b-28a652c3e20f","Type":"ContainerStarted","Data":"d4ddb869edcd19bf5a41a6525f4a863872794d30c33c07e4953c56721c81099a"}
Sep 11 16:17:13.681693 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:13.681661    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:13.682065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:13.681661    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:13.682065 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:13.681789    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:13.682065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:13.681661    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:13.682065 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:13.681869    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:13.682065 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:13.681960    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:13.941253 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:13.941168    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"54d7add73ced9b4325cd268cddf1a3236bf257f10a59f18b1314c8bcd3e5c485"}
Sep 11 16:17:15.681013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.680773    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:15.681815 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.680790    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:15.681815 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:15.681041    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:15.681815 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.680840    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:15.681815 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:15.681111    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:15.681815 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:15.681193    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:15.770946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.770919    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:17:15.771500 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.771483    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:17:15.947439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.947417    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="bc37ff0887c07f53aee98ecaec0d5fa63e70813a7cfba58032222dd940292c4e" exitCode=0
Sep 11 16:17:15.947549 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.947491    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"bc37ff0887c07f53aee98ecaec0d5fa63e70813a7cfba58032222dd940292c4e"}
Sep 11 16:17:15.951480 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.951454    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" event={"ID":"3e5c2285-c033-4362-abc2-0674294c8a2a","Type":"ContainerStarted","Data":"741af4040afa2680c62dfcdb99137a7c20f1d5fd79b16468c90fc3879ac1334c"}
Sep 11 16:17:15.952402 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.951654    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:15.952402 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.951731    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:15.952402 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.951784    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:15.954022 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.953996    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz" event={"ID":"598efbc0-8955-4220-8a1b-28a652c3e20f","Type":"ContainerStarted","Data":"7bebecadb5ff3410d8cb02ff146dde7f1f40bd99b09949a295eb9b70d16c9fdb"}
Sep 11 16:17:15.954571 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.954247    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:17:15.954716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.954690    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6qbgc"
Sep 11 16:17:15.967232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.967214    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:15.967787 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.967773    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:15.990110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:15.990071    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clhlz" podStartSLOduration=3.877491356 podStartE2EDuration="25.990060149s" totalImagesPullingTime="22.112568793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:15.989899374 +0000 UTC m=+26.764599954" watchObservedRunningTime="2026-09-11 16:17:15.990060149 +0000 UTC m=+26.764760726"
Sep 11 16:17:16.020864 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:16.020827    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5" podStartSLOduration=8.154129303 podStartE2EDuration="26.020815056s" totalImagesPullingTime="17.866685753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:16.020513476 +0000 UTC m=+26.795214054" watchObservedRunningTime="2026-09-11 16:17:16.020815056 +0000 UTC m=+26.795515634"
Sep 11 16:17:17.291279 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.290997    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wj4ht"]
Sep 11 16:17:17.291744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.291336    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:17.291744 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:17.291430    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:17.293736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.293707    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8pwdf"]
Sep 11 16:17:17.293887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.293816    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:17.293940 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:17.293910    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:17.294224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.294204    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6wc5s"]
Sep 11 16:17:17.294302 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.294292    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:17.294413 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:17.294394    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:17.960636 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.960587    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="f7c6a7b92fa81bbc2f823e95e36b2824ddd0912600bc856d11adedea6cd22946" exitCode=0
Sep 11 16:17:17.960803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:17.960655    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"f7c6a7b92fa81bbc2f823e95e36b2824ddd0912600bc856d11adedea6cd22946"}
Sep 11 16:17:18.680736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:18.680670    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:18.681060 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:18.680775    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:19.682067 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:19.681989    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:19.682470 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:19.682092    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:19.682470 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:19.682180    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:19.682470 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:19.682269    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:19.967799 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:19.967763    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="e22cf76cf88402298b6f33ffbe4d39e11f1a27fb039e039ff8089a2dd51f6cb9" exitCode=0
Sep 11 16:17:19.967958 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:19.967836    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"e22cf76cf88402298b6f33ffbe4d39e11f1a27fb039e039ff8089a2dd51f6cb9"}
Sep 11 16:17:20.681718 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:20.681613    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:20.681913 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:20.681825    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:21.681789 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:21.681675    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:21.681789 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:21.681709    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:21.682271 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:21.681805    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6wc5s" podUID="9a41d8fe-c043-4cf0-b634-afbb8447d967"
Sep 11 16:17:21.682271 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:21.681958    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wj4ht" podUID="c3698c03-eddd-48ee-882c-b8b7b3e9d9fc"
Sep 11 16:17:22.681673 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:22.681644    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:22.681832 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:22.681752    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pwdf" podUID="c0f02807-1ecb-4958-803a-c55a5246808a"
Sep 11 16:17:23.403583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.403546    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:23.403756 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.403590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:23.403756 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.403707    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:23.403756 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.403732    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:23.403880 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.403773    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.403752803 +0000 UTC m=+66.178453379 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:23.403880 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.403805    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret podName:9a41d8fe-c043-4cf0-b634-afbb8447d967 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.403786617 +0000 UTC m=+66.178487190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret") pod "global-pull-secret-syncer-6wc5s" (UID: "9a41d8fe-c043-4cf0-b634-afbb8447d967") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:23.504364 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.504336    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:23.504521 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.504497    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:23.504521 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.504520    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:23.504649 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.504530    2550 projected.go:196] Error preparing data for projected volume kube-api-access-kpckg for pod openshift-network-diagnostics/network-check-target-8pwdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:23.504649 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.504586    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg podName:c0f02807-1ecb-4958-803a-c55a5246808a nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.504569748 +0000 UTC m=+66.279270321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpckg" (UniqueName: "kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg") pod "network-check-target-8pwdf" (UID: "c0f02807-1ecb-4958-803a-c55a5246808a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:23.560662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.560636    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-175.ec2.internal" event="NodeReady"
Sep 11 16:17:23.560820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.560753    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:23.599360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.599330    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:17:23.601946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.601926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.604767 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.604701    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 16:17:23.604960 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.604940    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 16:17:23.605061 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.605040    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nwnk6\""
Sep 11 16:17:23.605123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.605088    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 16:17:23.612838 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.612818    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 16:17:23.622853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.622833    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nnkfw"]
Sep 11 16:17:23.624998 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.624981    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:17:23.625089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.625013    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2859b"]
Sep 11 16:17:23.625160 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.625111    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.626858 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.626840    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.628731 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.628712    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:23.628826 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.628745    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:23.631757 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.631736    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:23.631847 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.631786    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:23.631847 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.631779    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:23.631974 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.631883    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qbh4n\""
Sep 11 16:17:23.632029 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.631974    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:17:23.634767 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.634748    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nnkfw"]
Sep 11 16:17:23.643176 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.643151    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2859b"]
Sep 11 16:17:23.680996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.680973    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:23.681148 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.681130    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:23.684080 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.684062    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:23.684080 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.684087    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:23.684473 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.684087    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:17:23.705255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.705341 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705362    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbcw\" (UniqueName: \"kubernetes.io/projected/264a56a3-e3a2-4a4a-832d-2911100e725f-kube-api-access-6cbcw\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.705466 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705450    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-config-volume\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.705503 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705536 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705512    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nkx\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705553    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264a56a3-e3a2-4a4a-832d-2911100e725f-cert\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.705614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkg4w\" (UniqueName: \"kubernetes.io/projected/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-kube-api-access-rkg4w\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.705700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-tmp-dir\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.705700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705693    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705718    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705754    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.705806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.705789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.787764 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.787729    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:23.806591 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806565    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.806736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806604    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.806736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.806736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbcw\" (UniqueName: \"kubernetes.io/projected/264a56a3-e3a2-4a4a-832d-2911100e725f-kube-api-access-6cbcw\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.806886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.806743    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:23.806886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.806777    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:23.806886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.806794    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:23.806886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.806820    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:24.306801274 +0000 UTC m=+35.081501848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:23.806886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:23.806839    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:24.306828016 +0000 UTC m=+35.081528580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:23.807140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806911    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-config-volume\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.807140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.806941    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807052    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nkx\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807103    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264a56a3-e3a2-4a4a-832d-2911100e725f-cert\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.807140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807127    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkg4w\" (UniqueName: \"kubernetes.io/projected/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-kube-api-access-rkg4w\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807157    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-tmp-dir\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807208    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807232    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807266    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807265    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.807689 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-tmp-dir\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.807689 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807553    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-config-volume\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.807813 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.807793    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.808035 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.808010    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.811524 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.811504    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.811698 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.811676    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/264a56a3-e3a2-4a4a-832d-2911100e725f-cert\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.811766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.811681    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.820465 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.820327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbcw\" (UniqueName: \"kubernetes.io/projected/264a56a3-e3a2-4a4a-832d-2911100e725f-kube-api-access-6cbcw\") pod \"ingress-canary-nnkfw\" (UID: \"264a56a3-e3a2-4a4a-832d-2911100e725f\") " pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.821865 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.821844    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.823924 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.823892    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkg4w\" (UniqueName: \"kubernetes.io/projected/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-kube-api-access-rkg4w\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:23.825679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.825610    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nkx\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:23.937855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.937832    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nnkfw"
Sep 11 16:17:23.939078 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:23.939048    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:24.222988 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.222705    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nnkfw"]
Sep 11 16:17:24.225546 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:17:24.225514    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264a56a3_e3a2_4a4a_832d_2911100e725f.slice/crio-e6f3a417448b43fe7806c91e218ec63c10a72c2e36a0c417c8fa5d2f1a057e6f WatchSource:0}: Error finding container e6f3a417448b43fe7806c91e218ec63c10a72c2e36a0c417c8fa5d2f1a057e6f: Status 404 returned error can't find the container with id e6f3a417448b43fe7806c91e218ec63c10a72c2e36a0c417c8fa5d2f1a057e6f
Sep 11 16:17:24.273821 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.273791    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:24.311987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.311964    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:24.312063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.311990    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:24.312111 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:24.312098    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:24.312159 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:24.312112    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:24.312159 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:24.312101    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.312159 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:24.312157    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:25.312144421 +0000 UTC m=+36.086844977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:24.312269 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:24.312185    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:25.312165178 +0000 UTC m=+36.086865751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.680974 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.680901    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:24.684052 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.684019    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:24.684178 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.684050    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:24.685596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.685573    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:17:24.982657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.982604    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nnkfw" event={"ID":"264a56a3-e3a2-4a4a-832d-2911100e725f","Type":"ContainerStarted","Data":"e6f3a417448b43fe7806c91e218ec63c10a72c2e36a0c417c8fa5d2f1a057e6f"}
Sep 11 16:17:24.985746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.985717    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="45481f3621f8de79c97842e82c08f9c0aef3b0c10c1b770cf163af8f4b253679" exitCode=0
Sep 11 16:17:24.985870 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:24.985776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"45481f3621f8de79c97842e82c08f9c0aef3b0c10c1b770cf163af8f4b253679"}
Sep 11 16:17:25.318248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:25.318170    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:25.318248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:25.318212    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:25.318426 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:25.318340    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:25.318426 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:25.318389    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:25.318426 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:25.318402    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:25.318426 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:25.318415    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:27.318393502 +0000 UTC m=+38.093094070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:25.318581 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:25.318442    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:27.318431574 +0000 UTC m=+38.093132130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:25.991798 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:25.991765    2550 generic.go:356] "Generic (PLEG): container finished" podID="4c415c27-e7dd-4eae-a2ea-234ec2c85631" containerID="5c9d94b77c6b42770b5326e0361b0129bccc59e567dba8661474bd7f797ca2ab" exitCode=0
Sep 11 16:17:25.992275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:25.991832    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerDied","Data":"5c9d94b77c6b42770b5326e0361b0129bccc59e567dba8661474bd7f797ca2ab"}
Sep 11 16:17:26.757637 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:26.757376    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:26.857928 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:26.857894    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:26.957724 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:26.957696    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:26.995872 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:26.995848    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nnkfw" event={"ID":"264a56a3-e3a2-4a4a-832d-2911100e725f","Type":"ContainerStarted","Data":"de7cf404cd9ed673678751515f73f34474f126fe32e15d9c6802b8201e5c2a8b"}
Sep 11 16:17:26.999289 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:26.999270    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" event={"ID":"4c415c27-e7dd-4eae-a2ea-234ec2c85631","Type":"ContainerStarted","Data":"9ac68b95e5ea0fe4c5d6b4009bfb278ad112b1b47fcd3db2ad7a03691c46584e"}
Sep 11 16:17:27.017171 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.017109    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nnkfw" podStartSLOduration=1.937935977 podStartE2EDuration="4.01709781s" totalImagesPullingTime="2.079161833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:27.015917704 +0000 UTC m=+37.790618283" watchObservedRunningTime="2026-09-11 16:17:27.01709781 +0000 UTC m=+37.791798387"
Sep 11 16:17:27.042979 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.042938    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rgg4t" podStartSLOduration=8.87343923 podStartE2EDuration="37.042926014s" totalImagesPullingTime="28.169486784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:27.040871632 +0000 UTC m=+37.815572223" watchObservedRunningTime="2026-09-11 16:17:27.042926014 +0000 UTC m=+37.817626633"
Sep 11 16:17:27.162049 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.162024    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:27.245337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.245310    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:27.334877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.334817    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:27.334877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.334847    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:27.335030 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:27.334942    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:27.335030 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:27.334945    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:27.335030 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:27.334994    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:31.334980325 +0000 UTC m=+42.109680882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:27.335030 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:27.334952    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:27.335177 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:27.335041    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:31.335035111 +0000 UTC m=+42.109735666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:27.820242 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.820214    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:27.920388 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:27.920359    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.250687 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:28.250654    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.316468 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:28.316441    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.806585 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:28.806557    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:28.907019 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:28.906992    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:31.361834 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:31.361800    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:31.361834 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:31.361840    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:31.362306 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:31.361972    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:31.362306 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:31.362033    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.36201816 +0000 UTC m=+50.136718720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:31.362306 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:31.361979    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:31.362306 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:31.362068    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:31.362306 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:31.362111    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.362099797 +0000 UTC m=+50.136800372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:33.083295 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:33.083256    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.183261 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:33.183218    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:33.427641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:33.427594    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:36.452819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:36.452782    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:36.557023 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:36.556991    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:36.655393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:36.655362    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:36.787508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:36.787423    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:38.517348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:38.517315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:39.418994 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:39.418958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:39.418994 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:39.418991    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:39.419200 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:39.419109    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:39.419200 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:39.419123    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:39.419200 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:39.419167    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.419154365 +0000 UTC m=+66.193854921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:39.419200 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:39.419109    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:39.419332 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:39.419258    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.419242152 +0000 UTC m=+66.193942725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:39.977786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:39.977736    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:42.859226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:42.859193    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:42.927829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:42.927806    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.669552 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:43.669513    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:43.728084 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:43.728052    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:47.977509 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:47.977484    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rgj5"
Sep 11 16:17:48.152988 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:48.152957    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:48.252726 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:48.252659    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:55.434818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.434783    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:17:55.434818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.434822    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.434879    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.434908    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.434927    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.435007    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.43499273 +0000 UTC m=+98.209693289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.435008    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.435028    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:17:55.435365 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.435085    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.435069152 +0000 UTC m=+98.209769714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:17:55.437934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.437905    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:55.438070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.438017    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:55.445335 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.445317    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 16:17:55.445475 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:17:55.445368    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs podName:c3698c03-eddd-48ee-882c-b8b7b3e9d9fc nodeName:}" failed. No retries permitted until 2026-09-11 16:18:59.445357134 +0000 UTC m=+130.220057690 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs") pod "network-metrics-daemon-wj4ht" (UID: "c3698c03-eddd-48ee-882c-b8b7b3e9d9fc") : secret "metrics-daemon-secret" not found
Sep 11 16:17:55.448548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.448526    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a41d8fe-c043-4cf0-b634-afbb8447d967-original-pull-secret\") pod \"global-pull-secret-syncer-6wc5s\" (UID: \"9a41d8fe-c043-4cf0-b634-afbb8447d967\") " pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:55.492258 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.492238    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6wc5s"
Sep 11 16:17:55.535815 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.535793    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:55.538303 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.538287    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:55.549048 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.549028    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:55.559683 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.559659    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpckg\" (UniqueName: \"kubernetes.io/projected/c0f02807-1ecb-4958-803a-c55a5246808a-kube-api-access-kpckg\") pod \"network-check-target-8pwdf\" (UID: \"c0f02807-1ecb-4958-803a-c55a5246808a\") " pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:55.594544 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.594516    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:17:55.602684 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.602667    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:17:55.790818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.790794    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6wc5s"]
Sep 11 16:17:55.793650 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:17:55.793604    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a41d8fe_c043_4cf0_b634_afbb8447d967.slice/crio-23c1f0c3304a232837e1aebfdefb525008335b76ffb65e3b7f1e679fc8d7bf9f WatchSource:0}: Error finding container 23c1f0c3304a232837e1aebfdefb525008335b76ffb65e3b7f1e679fc8d7bf9f: Status 404 returned error can't find the container with id 23c1f0c3304a232837e1aebfdefb525008335b76ffb65e3b7f1e679fc8d7bf9f
Sep 11 16:17:55.896995 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:55.896857    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8pwdf"]
Sep 11 16:17:55.899186 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:17:55.899155    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f02807_1ecb_4958_803a_c55a5246808a.slice/crio-8a4898a12b3b42b84d6cec6fa9de4369b7bd9623b153c493547baacb5ef24817 WatchSource:0}: Error finding container 8a4898a12b3b42b84d6cec6fa9de4369b7bd9623b153c493547baacb5ef24817: Status 404 returned error can't find the container with id 8a4898a12b3b42b84d6cec6fa9de4369b7bd9623b153c493547baacb5ef24817
Sep 11 16:17:56.065438 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:56.065355    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8pwdf" event={"ID":"c0f02807-1ecb-4958-803a-c55a5246808a","Type":"ContainerStarted","Data":"8a4898a12b3b42b84d6cec6fa9de4369b7bd9623b153c493547baacb5ef24817"}
Sep 11 16:17:56.066562 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:56.066542    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6wc5s" event={"ID":"9a41d8fe-c043-4cf0-b634-afbb8447d967","Type":"ContainerStarted","Data":"23c1f0c3304a232837e1aebfdefb525008335b76ffb65e3b7f1e679fc8d7bf9f"}
Sep 11 16:17:58.415391 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:58.415355    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:58.519054 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:17:58.519013    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:01.080814 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:01.080771    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8pwdf" event={"ID":"c0f02807-1ecb-4958-803a-c55a5246808a","Type":"ContainerStarted","Data":"739959fd860f0c3dab9c17b29cdc4eb0be083966d8f03fe676e7495c0a3addb0"}
Sep 11 16:18:01.081307 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:01.080852    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:18:01.082217 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:01.082192    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6wc5s" event={"ID":"9a41d8fe-c043-4cf0-b634-afbb8447d967","Type":"ContainerStarted","Data":"1d7647931d113e16e258ff37ea7f0aeff0f865917726f26a7e029665a1a04f6d"}
Sep 11 16:18:01.100342 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:01.100301    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8pwdf" podStartSLOduration=66.811364672 podStartE2EDuration="1m11.100290274s" totalImagesPullingTime="4.288925602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:01.099388968 +0000 UTC m=+71.874089546" watchObservedRunningTime="2026-09-11 16:18:01.100290274 +0000 UTC m=+71.874990851"
Sep 11 16:18:01.114110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:01.114069    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6wc5s" podStartSLOduration=66.71662831 podStartE2EDuration="1m11.114053718s" totalImagesPullingTime="4.397425408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:01.11398042 +0000 UTC m=+71.888681001" watchObservedRunningTime="2026-09-11 16:18:01.114053718 +0000 UTC m=+71.888754298"
Sep 11 16:18:04.911010 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:04.910969    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.008089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:05.008057    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:19.730996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:19.730868    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:19.832352 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:19.832318    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.530774 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.530739    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.534588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.534564    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9b7c"]
Sep 11 16:18:23.537428 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.537405    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.540512 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.540489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:18:23.540646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.540522    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26bj7\""
Sep 11 16:18:23.540646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.540538    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:18:23.541652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.541639    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:18:23.541857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.541844    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:18:23.550200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.550180    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9b7c"]
Sep 11 16:18:23.613582 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.613554    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:23.621509 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.621489    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.621608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.621549    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nph59\" (UniqueName: \"kubernetes.io/projected/374640ff-5896-484e-97f3-2de2abae07e7-kube-api-access-nph59\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.621689 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.621616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/374640ff-5896-484e-97f3-2de2abae07e7-crio-socket\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.621744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.621683    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.621744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.621711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/374640ff-5896-484e-97f3-2de2abae07e7-data-volume\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.722848 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722824    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/374640ff-5896-484e-97f3-2de2abae07e7-crio-socket\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.722949 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722860    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.722949 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722878    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/374640ff-5896-484e-97f3-2de2abae07e7-data-volume\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.722949 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722909    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.722949 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722945    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nph59\" (UniqueName: \"kubernetes.io/projected/374640ff-5896-484e-97f3-2de2abae07e7-kube-api-access-nph59\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.723076 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.722946    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/374640ff-5896-484e-97f3-2de2abae07e7-crio-socket\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.723076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:23.722985    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.723076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:23.723007    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.723076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:23.723053    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.223032757 +0000 UTC m=+94.997733325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.723076 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:23.723067    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.223060286 +0000 UTC m=+94.997760842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.723264 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.723246    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/374640ff-5896-484e-97f3-2de2abae07e7-data-volume\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:23.732237 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:23.732210    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nph59\" (UniqueName: \"kubernetes.io/projected/374640ff-5896-484e-97f3-2de2abae07e7-kube-api-access-nph59\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:24.226946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:24.226905    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:24.227111 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:24.226953    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:24.227111 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:24.227026    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.227111 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:24.227049    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:24.227111 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:24.227090    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.22707178 +0000 UTC m=+96.001772358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.227111 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:24.227105    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.227098627 +0000 UTC m=+96.001799183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.233270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:25.233228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:25.233270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:25.233278    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:25.233720 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:25.233340    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.233720 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:25.233369    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.233720 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:25.233386    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.23337389 +0000 UTC m=+98.008074467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.233720 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:25.233421    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.23340816 +0000 UTC m=+98.008108719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.268832 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:25.268813    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bfjtf_29a5160d-5b16-4a1c-aa25-d3e03ef9a013/dns-node-resolver/0.log"
Sep 11 16:18:26.266243 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:26.266217    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccx2w_7379ca9e-d28c-4754-926d-f13edc395a9c/node-ca/0.log"
Sep 11 16:18:27.062167 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:27.062131    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nnkfw_264a56a3-e3a2-4a4a-832d-2911100e725f/serve-healthcheck-canary/0.log"
Sep 11 16:18:27.249612 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:27.249588    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:27.249767 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:27.249638    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:27.249767 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.249699    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.249767 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.249744    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.249731598 +0000 UTC m=+102.024432176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.249872 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.249781    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:27.249872 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.249846    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.249833979 +0000 UTC m=+102.024534548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:27.450904 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:27.450877    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:27.450908    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.451009    2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.451019    2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77fc874b44-g6g4r: secret "image-registry-tls" not found
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.451020    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.451063    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls podName:a2180806-0309-4f75-9196-5266cc1b3716 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:31.451050921 +0000 UTC m=+162.225751477 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls") pod "image-registry-77fc874b44-g6g4r" (UID: "a2180806-0309-4f75-9196-5266cc1b3716") : secret "image-registry-tls" not found
Sep 11 16:18:27.451294 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:27.451092    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls podName:7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:31.451076435 +0000 UTC m=+162.225776994 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls") pod "dns-default-2859b" (UID: "7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:29.541130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.541092    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"]
Sep 11 16:18:29.573292 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.573268    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"]
Sep 11 16:18:29.573431 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.573355    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"
Sep 11 16:18:29.576137 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.576117    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-n4xhl\""
Sep 11 16:18:29.576203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.576138    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:29.577162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.577141    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:29.667090 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.667059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmz8x\" (UniqueName: \"kubernetes.io/projected/f2947302-28e5-4bfd-871c-0b402161474c-kube-api-access-nmz8x\") pod \"volume-data-source-validator-86695b7cc7-dsqt5\" (UID: \"f2947302-28e5-4bfd-871c-0b402161474c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"
Sep 11 16:18:29.768338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.768311    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmz8x\" (UniqueName: \"kubernetes.io/projected/f2947302-28e5-4bfd-871c-0b402161474c-kube-api-access-nmz8x\") pod \"volume-data-source-validator-86695b7cc7-dsqt5\" (UID: \"f2947302-28e5-4bfd-871c-0b402161474c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"
Sep 11 16:18:29.777688 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.777661    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmz8x\" (UniqueName: \"kubernetes.io/projected/f2947302-28e5-4bfd-871c-0b402161474c-kube-api-access-nmz8x\") pod \"volume-data-source-validator-86695b7cc7-dsqt5\" (UID: \"f2947302-28e5-4bfd-871c-0b402161474c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"
Sep 11 16:18:29.882950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:29.882893    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"
Sep 11 16:18:30.172658 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:30.172570    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5"]
Sep 11 16:18:31.156215 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:31.156183    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5" event={"ID":"f2947302-28e5-4bfd-871c-0b402161474c","Type":"ContainerStarted","Data":"c5378eeb26a6581b84fbc96b480cda0e22a49340cdb0d0ff643cf59e2a107d98"}
Sep 11 16:18:31.279898 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:31.279864    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:31.279898 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:31.279910    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:31.280199 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:31.279979    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:31.280339 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:31.280315    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.280418 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:31.280406    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.280373585 +0000 UTC m=+110.055074156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.280486 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:31.280430    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.28041951 +0000 UTC m=+110.055120066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:32.087212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:32.087183    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8pwdf"
Sep 11 16:18:32.160912 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:32.160882    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5" event={"ID":"f2947302-28e5-4bfd-871c-0b402161474c","Type":"ContainerStarted","Data":"696cf8c2ec7ec7b4c24c6e1e91300ed7fcbb3eb499ac5cc9f441a6160d69c271"}
Sep 11 16:18:32.178193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:32.178146    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dsqt5" podStartSLOduration=1.72464487 podStartE2EDuration="3.178130529s" totalImagesPullingTime="1.453485659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:32.176972831 +0000 UTC m=+102.951673410" watchObservedRunningTime="2026-09-11 16:18:32.178130529 +0000 UTC m=+102.952831128"
Sep 11 16:18:32.280057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:32.280017    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.434707 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:32.434618    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.255133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.255100    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.324216 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.324192    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.703057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.703027    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.815175 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.815154    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"]
Sep 11 16:18:34.816951 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.816931    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:34.819043 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.819019    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:34.821055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.821037    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 16:18:34.821216 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.821204    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:18:34.821280 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.821262    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:18:34.821335 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.821276    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:18:34.830065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.830047    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"]
Sep 11 16:18:34.904138 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.904102    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-tmp\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:34.904306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.904192    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:34.904306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.904213    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckn2\" (UniqueName: \"kubernetes.io/projected/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-kube-api-access-8ckn2\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:34.954555 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:34.954476    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.005359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.005335    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.005484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.005363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckn2\" (UniqueName: \"kubernetes.io/projected/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-kube-api-access-8ckn2\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.005484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.005396    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-tmp\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.005854 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.005834    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-tmp\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.007998 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.007978    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.014746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.014723    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckn2\" (UniqueName: \"kubernetes.io/projected/5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5-kube-api-access-8ckn2\") pod \"klusterlet-addon-workmgr-54974d8c98-rszzd\" (UID: \"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.034330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.034294    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:35.126405 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.126382    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:35.391579 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:35.391558    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"]
Sep 11 16:18:35.393834 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:18:35.393804    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1b28c0_ba28_4ca9_9f91_bbbdc4ca0ce5.slice/crio-3595f436632748d39c4d1651769e44a03388816b266bb4eb1f484113e76b7e8b WatchSource:0}: Error finding container 3595f436632748d39c4d1651769e44a03388816b266bb4eb1f484113e76b7e8b: Status 404 returned error can't find the container with id 3595f436632748d39c4d1651769e44a03388816b266bb4eb1f484113e76b7e8b
Sep 11 16:18:36.172569 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.172530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd" event={"ID":"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5","Type":"ContainerStarted","Data":"3595f436632748d39c4d1651769e44a03388816b266bb4eb1f484113e76b7e8b"}
Sep 11 16:18:36.578968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.578935    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xw2bd"]
Sep 11 16:18:36.581119 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.581096    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.584768 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.584743    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 16:18:36.585257 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.585079    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 16:18:36.585257 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.585153    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 16:18:36.586164 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.586147    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-llz5r\""
Sep 11 16:18:36.586270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.586186    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 16:18:36.599695 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.599671    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xw2bd"]
Sep 11 16:18:36.617887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.617862    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/208a9aab-d796-42be-b4ef-9f278fea8128-signing-key\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.617887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.617897    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/208a9aab-d796-42be-b4ef-9f278fea8128-signing-cabundle\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.618096 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.617931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qz5\" (UniqueName: \"kubernetes.io/projected/208a9aab-d796-42be-b4ef-9f278fea8128-kube-api-access-v4qz5\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.651827 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.651794    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:36.719228 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.719197    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/208a9aab-d796-42be-b4ef-9f278fea8128-signing-key\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.719385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.719239    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/208a9aab-d796-42be-b4ef-9f278fea8128-signing-cabundle\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.719385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.719281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qz5\" (UniqueName: \"kubernetes.io/projected/208a9aab-d796-42be-b4ef-9f278fea8128-kube-api-access-v4qz5\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.720257 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.720225    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/208a9aab-d796-42be-b4ef-9f278fea8128-signing-cabundle\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.722142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.722114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/208a9aab-d796-42be-b4ef-9f278fea8128-signing-key\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.734435 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.734400    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qz5\" (UniqueName: \"kubernetes.io/projected/208a9aab-d796-42be-b4ef-9f278fea8128-kube-api-access-v4qz5\") pod \"service-ca-568869fdc5-xw2bd\" (UID: \"208a9aab-d796-42be-b4ef-9f278fea8128\") " pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:36.762162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.762012    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:36.893799 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:36.893714    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xw2bd"
Sep 11 16:18:37.744934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:37.744863    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xw2bd"]
Sep 11 16:18:38.472427 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:38.472389    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:38.817990 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:18:38.817927    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a9aab_d796_42be_b4ef_9f278fea8128.slice/crio-9deb4ae1d0cfc2cd86b6002d2769466fdcf270d6248954d32f16c53120a95ce6 WatchSource:0}: Error finding container 9deb4ae1d0cfc2cd86b6002d2769466fdcf270d6248954d32f16c53120a95ce6: Status 404 returned error can't find the container with id 9deb4ae1d0cfc2cd86b6002d2769466fdcf270d6248954d32f16c53120a95ce6
Sep 11 16:18:39.182836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.182759    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd" event={"ID":"5d1b28c0-ba28-4ca9-9f91-bbbdc4ca0ce5","Type":"ContainerStarted","Data":"746d9953e1eaf60a8f0cb9059aeb1ab32189f5257b24bd97639d5c320145f0b1"}
Sep 11 16:18:39.183001 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.182979    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:39.184157 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.184122    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xw2bd" event={"ID":"208a9aab-d796-42be-b4ef-9f278fea8128","Type":"ContainerStarted","Data":"9deb4ae1d0cfc2cd86b6002d2769466fdcf270d6248954d32f16c53120a95ce6"}
Sep 11 16:18:39.184602 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.184588    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd"
Sep 11 16:18:39.215844 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.215801    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54974d8c98-rszzd" podStartSLOduration=1.750440999 podStartE2EDuration="5.21578824s" totalImagesPullingTime="3.465347241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:39.215357524 +0000 UTC m=+109.990058101" watchObservedRunningTime="2026-09-11 16:18:39.21578824 +0000 UTC m=+109.990488864"
Sep 11 16:18:39.340956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.340923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:39.341161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.341134    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:39.341273 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:39.341255    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.341335 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:18:39.341324    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls podName:374640ff-5896-484e-97f3-2de2abae07e7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:55.341308481 +0000 UTC m=+126.116009378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9b7c" (UID: "374640ff-5896-484e-97f3-2de2abae07e7") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.341604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:39.341583    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/374640ff-5896-484e-97f3-2de2abae07e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:41.191437 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:41.191354    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xw2bd" event={"ID":"208a9aab-d796-42be-b4ef-9f278fea8128","Type":"ContainerStarted","Data":"7c49db31d53f6134dd84260e726f8566f5bc3e6306d70266f9f9afcf9b421b16"}
Sep 11 16:18:41.212025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:41.211982    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-xw2bd" podStartSLOduration=3.326387778 podStartE2EDuration="5.211968483s" totalImagesPullingTime="1.885580705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:41.210440959 +0000 UTC m=+111.985141538" watchObservedRunningTime="2026-09-11 16:18:41.211968483 +0000 UTC m=+111.986669062"
Sep 11 16:18:41.908343 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:41.908293    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.055538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:42.055500    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.161708 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:42.161610    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.262095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:42.262062    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:42.778723 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:42.778687    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.175402 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:43.175323    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:43.811505 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:43.811467    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.007027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:44.006989    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.237236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:44.237197    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.404614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:44.404583    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.604223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:44.604138    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:44.803887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:44.803857    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.790422 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:45.790379    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:45.865902 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:45.865873    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.016975 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:46.016944    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.186674 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:46.186584    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.404087 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:46.404051    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.486894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:46.486858    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.599674 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:46.599639    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.681586 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:47.681554    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.781917 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:47.781890    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.848949 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:47.848919    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:47.982112 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:47.982084    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.187054 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:48.187018    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.395987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:48.395909    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.471019 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:48.470984    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.587491 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:48.587449    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.604553 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:49.604527    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.725329 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:49.725297    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:49.981451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:49.981421    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:50.188372 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:50.188334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:50.378344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:50.378260    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:50.585581 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:50.585548    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:51.795368 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:51.795336    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:51.976589 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:51.976558    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:52.181782 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:52.181686    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:52.383956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:52.383922    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:52.582075 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:52.582037    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:53.387657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:53.387596    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:53.489732 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:53.489697    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:53.590311 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:53.590277    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:53.752885 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:53.752854    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:55.353637 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:55.353595    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:55.355903 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:55.355883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/374640ff-5896-484e-97f3-2de2abae07e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9b7c\" (UID: \"374640ff-5896-484e-97f3-2de2abae07e7\") " pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:55.648895 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:55.648827    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26bj7\""
Sep 11 16:18:55.656572 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:55.656553    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9b7c"
Sep 11 16:18:55.926516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:55.926446    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9b7c"]
Sep 11 16:18:55.930271 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:18:55.930247    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374640ff_5896_484e_97f3_2de2abae07e7.slice/crio-46c605743b4701b3027823ec6d0e2c2286a3814d423731f288fbf8339d33d2f5 WatchSource:0}: Error finding container 46c605743b4701b3027823ec6d0e2c2286a3814d423731f288fbf8339d33d2f5: Status 404 returned error can't find the container with id 46c605743b4701b3027823ec6d0e2c2286a3814d423731f288fbf8339d33d2f5
Sep 11 16:18:56.237215 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:56.237185    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9b7c" event={"ID":"374640ff-5896-484e-97f3-2de2abae07e7","Type":"ContainerStarted","Data":"8a8edd9c5053565adf8e043228b3f176cdbf1c85754b40a572c94159fc7f00f3"}
Sep 11 16:18:56.237350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:56.237235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9b7c" event={"ID":"374640ff-5896-484e-97f3-2de2abae07e7","Type":"ContainerStarted","Data":"46c605743b4701b3027823ec6d0e2c2286a3814d423731f288fbf8339d33d2f5"}
Sep 11 16:18:57.245058 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:57.245022    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9b7c" event={"ID":"374640ff-5896-484e-97f3-2de2abae07e7","Type":"ContainerStarted","Data":"7cfa03609387e234062eca21d82c92434d615f3ab791a36c1f6f38b31aff41f6"}
Sep 11 16:18:58.250119 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:58.250080    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9b7c" event={"ID":"374640ff-5896-484e-97f3-2de2abae07e7","Type":"ContainerStarted","Data":"3ed321e06c667b979c677244dd5a41ab9f4f3251b29d0fa01beffd20b6ca160d"}
Sep 11 16:18:58.270772 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:58.270728    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b9b7c" podStartSLOduration=33.429510846 podStartE2EDuration="35.270713399s" totalImagesPullingTime="1.841202553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:58.269559464 +0000 UTC m=+129.044260043" watchObservedRunningTime="2026-09-11 16:18:58.270713399 +0000 UTC m=+129.045413976"
Sep 11 16:18:58.767890 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:58.767858    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:58.907219 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:58.907192    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.036089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.036023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:59.483970 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.483946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:18:59.486156 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.486135    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3698c03-eddd-48ee-882c-b8b7b3e9d9fc-metrics-certs\") pod \"network-metrics-daemon-wj4ht\" (UID: \"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc\") " pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:18:59.700527 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.700502    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:18:59.707807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.707790    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wj4ht"
Sep 11 16:18:59.972113 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:18:59.972084    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wj4ht"]
Sep 11 16:18:59.974806 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:18:59.974774    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3698c03_eddd_48ee_882c_b8b7b3e9d9fc.slice/crio-f1e8d16815b7a02258344ec45ba1eecf9f0560da78881773749fd74520abc165 WatchSource:0}: Error finding container f1e8d16815b7a02258344ec45ba1eecf9f0560da78881773749fd74520abc165: Status 404 returned error can't find the container with id f1e8d16815b7a02258344ec45ba1eecf9f0560da78881773749fd74520abc165
Sep 11 16:19:00.256898 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:00.256866    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wj4ht" event={"ID":"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc","Type":"ContainerStarted","Data":"f1e8d16815b7a02258344ec45ba1eecf9f0560da78881773749fd74520abc165"}
Sep 11 16:19:01.261741 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.261702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wj4ht" event={"ID":"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc","Type":"ContainerStarted","Data":"bee41ea8358f19a2384f444cf9ff6ddbafd0622ec18b4e56b30cd8bf425ea510"}
Sep 11 16:19:01.262117 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.261777    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wj4ht" event={"ID":"c3698c03-eddd-48ee-882c-b8b7b3e9d9fc","Type":"ContainerStarted","Data":"3d7ce208a7f6fc2653addf3e1d5a36634f462af8d0508a7a7956aab1d4494516"}
Sep 11 16:19:01.281602 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.281557    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wj4ht" podStartSLOduration=130.498772852 podStartE2EDuration="2m11.281542307s" totalImagesPullingTime="782.769455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:01.280147857 +0000 UTC m=+132.054848434" watchObservedRunningTime="2026-09-11 16:19:01.281542307 +0000 UTC m=+132.056242885"
Sep 11 16:19:01.388220 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.388195    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:01.464218 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.464195    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:01.532931 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.532876    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:01.722538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.722507    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:01.824814 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.824726    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:01.883259 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:01.883231    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.621744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:02.621712    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.695373 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:02.695346    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.952114 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:02.952084    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:03.264232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:03.264152    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.565252 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:04.565213    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:05.787525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:05.787492    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.058387 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.058308    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.081948 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.081919    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"]
Sep 11 16:19:06.084204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.084183    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.088906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.088886    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:19:06.090172 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.090081    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d6rtz\""
Sep 11 16:19:06.090172 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.090136    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:19:06.090172 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.090163    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:19:06.090388 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.090136    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 16:19:06.090388 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.090338    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:19:06.106328 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.106310    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"]
Sep 11 16:19:06.108359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.108343    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"]
Sep 11 16:19:06.108475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.108458    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.112556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.112119    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:19:06.112556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.112133    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 16:19:06.112556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.112214    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rfb6k\""
Sep 11 16:19:06.112556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.112125    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 16:19:06.128122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.128102    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"]
Sep 11 16:19:06.143953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.143932    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nwnps"]
Sep 11 16:19:06.146468 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.146452    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.149503 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.149484    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2bzxn\""
Sep 11 16:19:06.149850 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.149804    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:19:06.150079 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.150065    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:19:06.150079 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.150072    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:19:06.168796 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.168756    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.227508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227482    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-tls\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.227611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227513    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5074fd5a-de72-47a6-b80c-48480094d971-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.227611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227542    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.227611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227567    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c27878f-25c4-4010-9cd9-a4c38bb1b394-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.227611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227594    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.227822 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227642    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgdj\" (UniqueName: \"kubernetes.io/projected/0c27878f-25c4-4010-9cd9-a4c38bb1b394-kube-api-access-5bgdj\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.227822 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227674    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.227822 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227734    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-sys\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.227822 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-textfile\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.227961 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227824    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-root\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.227961 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227848    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-wtmp\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.227961 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.228065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.228065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.227988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zh9\" (UniqueName: \"kubernetes.io/projected/5074fd5a-de72-47a6-b80c-48480094d971-kube-api-access-r8zh9\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.228065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.228019    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.228065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.228045    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-metrics-client-ca\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.228224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.228071    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.228224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.228124    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.228224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.228173    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/256e8b72-ac3e-4292-9484-57f0c8e03b47-kube-api-access-psqxd\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.263494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.263470    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.329276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329211    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgdj\" (UniqueName: \"kubernetes.io/projected/0c27878f-25c4-4010-9cd9-a4c38bb1b394-kube-api-access-5bgdj\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.329276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.329276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329259    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-sys\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-textfile\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329301    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-root\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329325    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-wtmp\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329362    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-sys\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329425    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-root\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329455    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-wtmp\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329455    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329515    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329546 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329542    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zh9\" (UniqueName: \"kubernetes.io/projected/5074fd5a-de72-47a6-b80c-48480094d971-kube-api-access-r8zh9\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329579    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329608    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-metrics-client-ca\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329663    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329669    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-textfile\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329716    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329750    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/256e8b72-ac3e-4292-9484-57f0c8e03b47-kube-api-access-psqxd\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-tls\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5074fd5a-de72-47a6-b80c-48480094d971-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329870    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329898    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c27878f-25c4-4010-9cd9-a4c38bb1b394-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.329969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.329923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.330500 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.330024    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-accelerators-collector-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.330688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.330762    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/256e8b72-ac3e-4292-9484-57f0c8e03b47-metrics-client-ca\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:19:06.330853    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:19:06.330925    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls podName:0c27878f-25c4-4010-9cd9-a4c38bb1b394 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:06.830889089 +0000 UTC m=+137.605589660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-28hm5" (UID: "0c27878f-25c4-4010-9cd9-a4c38bb1b394") : secret "openshift-state-metrics-tls" not found
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.331021    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5074fd5a-de72-47a6-b80c-48480094d971-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.331161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.331139    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.331683 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.331658    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c27878f-25c4-4010-9cd9-a4c38bb1b394-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.332661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.332637    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.332925 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.332905    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.333288 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.333266    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5074fd5a-de72-47a6-b80c-48480094d971-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.333339 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.333283    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/256e8b72-ac3e-4292-9484-57f0c8e03b47-node-exporter-tls\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.334375 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.334354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.350813 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.350263    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zh9\" (UniqueName: \"kubernetes.io/projected/5074fd5a-de72-47a6-b80c-48480094d971-kube-api-access-r8zh9\") pod \"kube-state-metrics-59fbdd77d-4qsjw\" (UID: \"5074fd5a-de72-47a6-b80c-48480094d971\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.351098 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.351077    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/256e8b72-ac3e-4292-9484-57f0c8e03b47-kube-api-access-psqxd\") pod \"node-exporter-nwnps\" (UID: \"256e8b72-ac3e-4292-9484-57f0c8e03b47\") " pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.351193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.351171    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgdj\" (UniqueName: \"kubernetes.io/projected/0c27878f-25c4-4010-9cd9-a4c38bb1b394-kube-api-access-5bgdj\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.361780 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.361752    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.419837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.419804    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"
Sep 11 16:19:06.457126 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.457093    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nwnps"
Sep 11 16:19:06.469200 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:06.469104    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256e8b72_ac3e_4292_9484_57f0c8e03b47.slice/crio-87619f8e1eac26de7c5694b829febc139af808a1fe461cb7b6c08a408371ef1f WatchSource:0}: Error finding container 87619f8e1eac26de7c5694b829febc139af808a1fe461cb7b6c08a408371ef1f: Status 404 returned error can't find the container with id 87619f8e1eac26de7c5694b829febc139af808a1fe461cb7b6c08a408371ef1f
Sep 11 16:19:06.539306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.539267    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.633955 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.633882    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.702266 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.702244    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw"]
Sep 11 16:19:06.704426 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:06.704403    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5074fd5a_de72_47a6_b80c_48480094d971.slice/crio-f04e9c27e8452f04b4fb1adf2eba66b5046df471abc0aef3421e3d0cefc3f24a WatchSource:0}: Error finding container f04e9c27e8452f04b4fb1adf2eba66b5046df471abc0aef3421e3d0cefc3f24a: Status 404 returned error can't find the container with id f04e9c27e8452f04b4fb1adf2eba66b5046df471abc0aef3421e3d0cefc3f24a
Sep 11 16:19:06.712076 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.712045    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:06.834857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.834823    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.837068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.837050    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c27878f-25c4-4010-9cd9-a4c38bb1b394-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-28hm5\" (UID: \"0c27878f-25c4-4010-9cd9-a4c38bb1b394\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:06.995574 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:06.995543    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"
Sep 11 16:19:07.024024 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.023990    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:07.100493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.100464    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:07.112369 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.112345    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:07.115817 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.115799    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.118568 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118544    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:19:07.118568 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118558    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:19:07.118800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118569    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:19:07.118800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118575    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:19:07.118800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118575    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:19:07.118957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118820    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:19:07.118957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118822    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9sj25\""
Sep 11 16:19:07.118957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118895    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:19:07.118957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118822    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:19:07.118957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.118824    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:19:07.137940 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.137917    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.238846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.238889    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.238924    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.238955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.238980    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239052    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239105    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239172    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239197    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.239416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.239224    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.274606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.272958    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5"]
Sep 11 16:19:07.276459 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:07.276431    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c27878f_25c4_4010_9cd9_a4c38bb1b394.slice/crio-cc5e87f0d110144848c531d1c3b0b8536a4d74c89b6bc16c0e8a3581b7faa3e1 WatchSource:0}: Error finding container cc5e87f0d110144848c531d1c3b0b8536a4d74c89b6bc16c0e8a3581b7faa3e1: Status 404 returned error can't find the container with id cc5e87f0d110144848c531d1c3b0b8536a4d74c89b6bc16c0e8a3581b7faa3e1
Sep 11 16:19:07.282596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.282543    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw" event={"ID":"5074fd5a-de72-47a6-b80c-48480094d971","Type":"ContainerStarted","Data":"f04e9c27e8452f04b4fb1adf2eba66b5046df471abc0aef3421e3d0cefc3f24a"}
Sep 11 16:19:07.284507 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.284480    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwnps" event={"ID":"256e8b72-ac3e-4292-9484-57f0c8e03b47","Type":"ContainerStarted","Data":"87619f8e1eac26de7c5694b829febc139af808a1fe461cb7b6c08a408371ef1f"}
Sep 11 16:19:07.340238 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340211    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340255    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340304    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340329    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340352    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340383    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340451    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340476    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340776 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340505    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340776 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340587    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340776 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340613    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.340776 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.340689    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.341885 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.341533    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.342267 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.342243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.343139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.343114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.345267 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.345241    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.346412 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.346234    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.346412 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.346373    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.349381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.348226    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.349381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.348337    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.349381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.348735    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.349647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.349382    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.351675 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.349775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.351675 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.350908    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.353298 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.353255    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8\") pod \"alertmanager-main-0\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:07.369135 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.369079    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:07.425538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:07.425511    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:08.253691 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.253657    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:08.257589 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:08.257556    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1062c954_c2d5_4a49_9dab_f744b7c67215.slice/crio-3469a8927358dfab1549966fda91a1348c474a9f3d9841ac69daeb7b2cbf8839 WatchSource:0}: Error finding container 3469a8927358dfab1549966fda91a1348c474a9f3d9841ac69daeb7b2cbf8839: Status 404 returned error can't find the container with id 3469a8927358dfab1549966fda91a1348c474a9f3d9841ac69daeb7b2cbf8839
Sep 11 16:19:08.295680 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.295651    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5" event={"ID":"0c27878f-25c4-4010-9cd9-a4c38bb1b394","Type":"ContainerStarted","Data":"ae153c9d36897f2557b4f9fb56835d97209aaed3b9c9cf7e21b3b4dc30f62be6"}
Sep 11 16:19:08.295943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.295906    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5" event={"ID":"0c27878f-25c4-4010-9cd9-a4c38bb1b394","Type":"ContainerStarted","Data":"9fb2c46656d267acac2d16fa165c2fcb4d65a6a55f29c286c78c58301e61796e"}
Sep 11 16:19:08.296048 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.295948    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5" event={"ID":"0c27878f-25c4-4010-9cd9-a4c38bb1b394","Type":"ContainerStarted","Data":"cc5e87f0d110144848c531d1c3b0b8536a4d74c89b6bc16c0e8a3581b7faa3e1"}
Sep 11 16:19:08.298533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.298505    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw" event={"ID":"5074fd5a-de72-47a6-b80c-48480094d971","Type":"ContainerStarted","Data":"af017cfcb0423108f7b856d3ca208981d124d55f5d08c07a7f3250d66535b624"}
Sep 11 16:19:08.298652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.298566    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw" event={"ID":"5074fd5a-de72-47a6-b80c-48480094d971","Type":"ContainerStarted","Data":"fc31dbf5c4cf68b9081617df18fca4a31923651e5d13c11acccf968f2c1092f7"}
Sep 11 16:19:08.298652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.298594    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw" event={"ID":"5074fd5a-de72-47a6-b80c-48480094d971","Type":"ContainerStarted","Data":"24b92a8a8ae88d68d0b5b61fc380e7f9cce1806cc622b8d2ca49ce84b0ef21ab"}
Sep 11 16:19:08.300434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.300382    2550 generic.go:356] "Generic (PLEG): container finished" podID="256e8b72-ac3e-4292-9484-57f0c8e03b47" containerID="e470b88eb46566fcf74932b2f31c12a01a757797cec3f45867f04d5aceeb5120" exitCode=0
Sep 11 16:19:08.300543 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.300458    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwnps" event={"ID":"256e8b72-ac3e-4292-9484-57f0c8e03b47","Type":"ContainerDied","Data":"e470b88eb46566fcf74932b2f31c12a01a757797cec3f45867f04d5aceeb5120"}
Sep 11 16:19:08.301910 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.301888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"3469a8927358dfab1549966fda91a1348c474a9f3d9841ac69daeb7b2cbf8839"}
Sep 11 16:19:08.353819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.353770    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4qsjw" podStartSLOduration=1.087318427 podStartE2EDuration="2.353754544s" totalImagesPullingTime="1.266436117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:08.327374893 +0000 UTC m=+139.102075473" watchObservedRunningTime="2026-09-11 16:19:08.353754544 +0000 UTC m=+139.128455121"
Sep 11 16:19:08.998837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:08.998760    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:09.099958 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.099925    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:09.199811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.199774    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:09.300462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.300434    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:09.306826 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.306803    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwnps" event={"ID":"256e8b72-ac3e-4292-9484-57f0c8e03b47","Type":"ContainerStarted","Data":"26649e145f28e6cf83afff58e9843619142bbf0870d3cf7a0ae9fd605cf7c7da"}
Sep 11 16:19:09.306930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.306862    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nwnps" event={"ID":"256e8b72-ac3e-4292-9484-57f0c8e03b47","Type":"ContainerStarted","Data":"d36008f3156d511c0fdea4a32299732fbbb236d63ff4e9a9e77feff4d20e0305"}
Sep 11 16:19:09.309075 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.309052    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5" event={"ID":"0c27878f-25c4-4010-9cd9-a4c38bb1b394","Type":"ContainerStarted","Data":"e7dea1149b9d949bbbe3846f4e55a8f7dcdc586fc04b2df987f327e84962c689"}
Sep 11 16:19:09.356042 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.355999    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nwnps" podStartSLOduration=1.767792077 podStartE2EDuration="3.355983425s" totalImagesPullingTime="1.500210641s" totalInitContainerRuntime="87.980707ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:09.332648801 +0000 UTC m=+140.107349376" watchObservedRunningTime="2026-09-11 16:19:09.355983425 +0000 UTC m=+140.130684356"
Sep 11 16:19:09.356421 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:09.356393    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-28hm5" podStartSLOduration=2.196086345 podStartE2EDuration="3.356384985s" totalImagesPullingTime="1.16029864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:09.355316243 +0000 UTC m=+140.130016821" watchObservedRunningTime="2026-09-11 16:19:09.356384985 +0000 UTC m=+140.131085562"
Sep 11 16:19:10.314176 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.314146    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="a5f31398d3a73dc18fa31fe33b21d0bb1dbe8a193aa333b73e53ef9138bdef4f" exitCode=0
Sep 11 16:19:10.314538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.314235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"a5f31398d3a73dc18fa31fe33b21d0bb1dbe8a193aa333b73e53ef9138bdef4f"}
Sep 11 16:19:10.376365 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.376339    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5949df56d6-gvqpx"]
Sep 11 16:19:10.401710 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.401684    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5949df56d6-gvqpx"]
Sep 11 16:19:10.401837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.401764    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.404763 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.404739    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 16:19:10.404881 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.404801    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g6vgq\""
Sep 11 16:19:10.404881 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.404746    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 16:19:10.404881 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.404746    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f1l797refvqn2\""
Sep 11 16:19:10.404881 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.404853    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 16:19:10.406001 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.405986    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 16:19:10.570690 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570607    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-client-certs\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570733    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9accd2e3-d2ee-403c-a246-8684f7c01044-audit-log\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570752    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wtm\" (UniqueName: \"kubernetes.io/projected/9accd2e3-d2ee-403c-a246-8684f7c01044-kube-api-access-g2wtm\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-client-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570807    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-tls\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.570952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.570831    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-metrics-server-audit-profiles\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671535 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671507    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671535 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671539    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9accd2e3-d2ee-403c-a246-8684f7c01044-audit-log\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wtm\" (UniqueName: \"kubernetes.io/projected/9accd2e3-d2ee-403c-a246-8684f7c01044-kube-api-access-g2wtm\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671690    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-client-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671775 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671748    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-tls\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671787    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-metrics-server-audit-profiles\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671874    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9accd2e3-d2ee-403c-a246-8684f7c01044-audit-log\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.671948 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.671923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-client-certs\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.672216 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.672196    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.672587 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.672566    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9accd2e3-d2ee-403c-a246-8684f7c01044-metrics-server-audit-profiles\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.674330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.674308    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-client-certs\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.674410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.674364    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-secret-metrics-server-tls\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.674410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.674381    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9accd2e3-d2ee-403c-a246-8684f7c01044-client-ca-bundle\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.679997 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.679975    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wtm\" (UniqueName: \"kubernetes.io/projected/9accd2e3-d2ee-403c-a246-8684f7c01044-kube-api-access-g2wtm\") pod \"metrics-server-5949df56d6-gvqpx\" (UID: \"9accd2e3-d2ee-403c-a246-8684f7c01044\") " pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.711089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.711071    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:10.714828 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.714803    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:10.816009 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.815975    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:10.899467 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.899397    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:10.958028 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.957994    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:10.997463 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:10.997434    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5949df56d6-gvqpx"]
Sep 11 16:19:11.001267 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:11.001237    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9accd2e3_d2ee_403c_a246_8684f7c01044.slice/crio-f37eef49b06fc73ce39077803582d0348f70100bd1fd476332a341ac5a010b98 WatchSource:0}: Error finding container f37eef49b06fc73ce39077803582d0348f70100bd1fd476332a341ac5a010b98: Status 404 returned error can't find the container with id f37eef49b06fc73ce39077803582d0348f70100bd1fd476332a341ac5a010b98
Sep 11 16:19:11.314946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.314913    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-684c788d95-h7dzh"]
Sep 11 16:19:11.338214 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.338178    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx" event={"ID":"9accd2e3-d2ee-403c-a246-8684f7c01044","Type":"ContainerStarted","Data":"f37eef49b06fc73ce39077803582d0348f70100bd1fd476332a341ac5a010b98"}
Sep 11 16:19:11.338214 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.338217    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-684c788d95-h7dzh"]
Sep 11 16:19:11.338427 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.338338    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.342460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.342433    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 16:19:11.342816 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.342790    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-98btb\""
Sep 11 16:19:11.342926 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.342796    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 16:19:11.343189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.343171    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 16:19:11.343243 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.343195    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 16:19:11.343298 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.343242    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 16:19:11.348784 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.348765    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 16:19:11.480095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.479991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480065    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7s8\" (UniqueName: \"kubernetes.io/projected/610aabb3-ea79-426e-afee-9a10c429506a-kube-api-access-7r7s8\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480203    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480250    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-serving-certs-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-federate-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480354    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.480548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.480420    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-metrics-client-ca\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581032 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.580938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-metrics-client-ca\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581032 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.580993    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581032 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581033    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7s8\" (UniqueName: \"kubernetes.io/projected/610aabb3-ea79-426e-afee-9a10c429506a-kube-api-access-7r7s8\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581057    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581094    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581142    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-serving-certs-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581169    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-federate-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581189    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.581901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.581867    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-metrics-client-ca\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.582196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.582144    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-serving-certs-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.582353 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.582329    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.584158 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.584120    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.584282 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.584262    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.584374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.584352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-telemeter-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.584607 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.584589    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/610aabb3-ea79-426e-afee-9a10c429506a-federate-client-tls\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.590963 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.590945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7s8\" (UniqueName: \"kubernetes.io/projected/610aabb3-ea79-426e-afee-9a10c429506a-kube-api-access-7r7s8\") pod \"telemeter-client-684c788d95-h7dzh\" (UID: \"610aabb3-ea79-426e-afee-9a10c429506a\") " pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:11.648027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:11.648000    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh"
Sep 11 16:19:12.019184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.019158    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-684c788d95-h7dzh"]
Sep 11 16:19:12.076100 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:12.076077    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610aabb3_ea79_426e_afee_9a10c429506a.slice/crio-2fdeb45c9f4697eaa5e8253a050df01bfdc3f2dcd53d5d2827998cb8050ffb19 WatchSource:0}: Error finding container 2fdeb45c9f4697eaa5e8253a050df01bfdc3f2dcd53d5d2827998cb8050ffb19: Status 404 returned error can't find the container with id 2fdeb45c9f4697eaa5e8253a050df01bfdc3f2dcd53d5d2827998cb8050ffb19
Sep 11 16:19:12.303500 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.303311    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:12.304179 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.304154    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:12.308029 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.307646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.310363 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.310324    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:19:12.311658 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.311502    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313491    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313578    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313594    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313735    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313809    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313836    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:19:12.313996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313941    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:19:12.314382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.314027    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:19:12.315128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.314588    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:19:12.315128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.313425    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6k6vvudrnq33p\""
Sep 11 16:19:12.315128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.314894    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:19:12.315572 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.315183    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7g5df\""
Sep 11 16:19:12.318679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.318539    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:19:12.330667 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.330640    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:12.333175 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.333150    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:19:12.334111 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.334087    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh" event={"ID":"610aabb3-ea79-426e-afee-9a10c429506a","Type":"ContainerStarted","Data":"2fdeb45c9f4697eaa5e8253a050df01bfdc3f2dcd53d5d2827998cb8050ffb19"}
Sep 11 16:19:12.337034 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.337012    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"0d457063fb5072783c4fe3815eb17a01d82b79bb6a5482369ed2c0352b878db4"}
Sep 11 16:19:12.337200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.337186    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"4ff01030cd97864e5bb28d51d6d0b32213b1739e852b4519c0779a3408c7b1e8"}
Sep 11 16:19:12.388049 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388020    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388197 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388180    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388299 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388283    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8wn\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388390    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388488 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388475    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388634 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388748    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388857    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.388968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.388955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389146 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389132    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389290 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389276    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389463    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389589 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389577    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389721 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389706    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389840 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389823    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.389946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.389932    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.390054 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.390038    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.390159 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.390145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.390256 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.390243    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491043    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491073    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491090    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491107    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8wn\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491126    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491141    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491164    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491180    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491199    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491213    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491234    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491261    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491275    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491309    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491325    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491348    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491366    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.491447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491390    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.492060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.491969    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.492667 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.492650    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.493607 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.493366    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.494807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.493928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.494807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.494476    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.495223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.495200    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.495977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.496083    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.496591    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.496716    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497561 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.497421    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.497696    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.497796    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.497941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.497894    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.498566 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.498520    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.498566 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.498562    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.498901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.498881    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.500047 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.500004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.500133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.500050    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.501933 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.501911    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8wn\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn\") pod \"prometheus-k8s-0\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:12.621592 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:12.621499    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:13.277017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.276932    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:13.321234 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:13.321202    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe3cb50_ec96_4695_bdc6_0cfaef9eb63d.slice/crio-0a9038ac452d65c4229013e040d1cb26e1a67807692a7094aef427de9208d08b WatchSource:0}: Error finding container 0a9038ac452d65c4229013e040d1cb26e1a67807692a7094aef427de9208d08b: Status 404 returned error can't find the container with id 0a9038ac452d65c4229013e040d1cb26e1a67807692a7094aef427de9208d08b
Sep 11 16:19:13.342817 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.342788    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"0a9038ac452d65c4229013e040d1cb26e1a67807692a7094aef427de9208d08b"}
Sep 11 16:19:13.346137 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.346108    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"ca1fb9c2561dcc5f7347604b30075899a94a62a2779842d8c2056e93e794f06b"}
Sep 11 16:19:13.346266 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.346170    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"b0601d48feb74a6db73be9dd2016918e4cd1a1d87a541a75840fb9659ee21544"}
Sep 11 16:19:13.346266 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.346199    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"2355465a1e746dddacaa9cfcafa8ee3c7234abddf8d4d6da7ac9eddd81bcf907"}
Sep 11 16:19:13.347802 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.347773    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx" event={"ID":"9accd2e3-d2ee-403c-a246-8684f7c01044","Type":"ContainerStarted","Data":"c4fc5a42bd415fab1e749f39064ea11d308a64062c7c3b3724f9db722b2ce3a5"}
Sep 11 16:19:13.367203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:13.367151    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx" podStartSLOduration=1.355140965 podStartE2EDuration="3.367128777s" totalImagesPullingTime="2.011987812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:13.366127435 +0000 UTC m=+144.140828013" watchObservedRunningTime="2026-09-11 16:19:13.367128777 +0000 UTC m=+144.141829357"
Sep 11 16:19:14.353830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.353789    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh" event={"ID":"610aabb3-ea79-426e-afee-9a10c429506a","Type":"ContainerStarted","Data":"4ffe1404c46c1abf37641cac30cdfe4a58f240925a5ffb0d23302d84457637fa"}
Sep 11 16:19:14.354254 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.353895    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh" event={"ID":"610aabb3-ea79-426e-afee-9a10c429506a","Type":"ContainerStarted","Data":"4c83bb49b0fbcccd944ec302260b1f72cf75e8ec0cf5df05912a666851862f8a"}
Sep 11 16:19:14.354254 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.353919    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh" event={"ID":"610aabb3-ea79-426e-afee-9a10c429506a","Type":"ContainerStarted","Data":"a04cb6bd4938dc213d11371e718c7edf37dba90fd50117afee9e043d820aa91f"}
Sep 11 16:19:14.355335 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.355310    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e" exitCode=0
Sep 11 16:19:14.355439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.355391    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"}
Sep 11 16:19:14.358778 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.358756    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerStarted","Data":"a737f8c4bcae35333a2db6a2ef8710fa6ac8a785b549c85974d4a70bc41866b7"}
Sep 11 16:19:14.385744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.385702    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-684c788d95-h7dzh" podStartSLOduration=1.683117603 podStartE2EDuration="3.385688805s" totalImagesPullingTime="1.702571202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:14.3838071 +0000 UTC m=+145.158507679" watchObservedRunningTime="2026-09-11 16:19:14.385688805 +0000 UTC m=+145.160389383"
Sep 11 16:19:14.442786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:14.442743    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.329621847 podStartE2EDuration="7.442727629s" totalImagesPullingTime="4.089720642s" totalInitContainerRuntime="23.38514ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:14.441390366 +0000 UTC m=+145.216090986" watchObservedRunningTime="2026-09-11 16:19:14.442727629 +0000 UTC m=+145.217428207"
Sep 11 16:19:17.142771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:17.142710    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:17.242281 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:17.242241    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:18.380950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:18.380911    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"}
Sep 11 16:19:18.381334 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:18.380980    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"}
Sep 11 16:19:20.393961 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:20.393927    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"}
Sep 11 16:19:20.394344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:20.394015    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"}
Sep 11 16:19:20.394344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:20.394048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"}
Sep 11 16:19:20.394344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:20.394073    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerStarted","Data":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"}
Sep 11 16:19:20.431190 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:20.431124    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.2854786320000002 podStartE2EDuration="8.431102962s" totalImagesPullingTime="5.116863861s" totalInitContainerRuntime="28.760469ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:20.425459307 +0000 UTC m=+151.200159886" watchObservedRunningTime="2026-09-11 16:19:20.431102962 +0000 UTC m=+151.205803542"
Sep 11 16:19:21.579678 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:21.579645    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:21.648841 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:21.648813    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:22.120785 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:22.120756    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:22.222959 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:22.222934    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:22.622415 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:22.622391    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:26.613783 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:19:26.613724    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" podUID="a2180806-0309-4f75-9196-5266cc1b3716"
Sep 11 16:19:26.644964 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:19:26.644932    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2859b" podUID="7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663"
Sep 11 16:19:27.422811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:27.422784    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:27.422975 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:27.422784    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:28.539603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:28.539571    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:28.636212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:28.636160    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:28.705233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:28.705200    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:28.821362 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:28.821276    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.208845 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.208811    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.288122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.288088    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.400327 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.400277    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.466855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.466753    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.606237 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.606215    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.682083 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:29.682046    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:30.008904 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:30.008855    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:30.122591 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:30.122565    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:30.223818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:30.223786    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:30.712153 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:30.712124    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:30.712489 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:30.712176    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:31.462480 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.462445    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:31.462677 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.462505    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:31.464803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.464772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663-metrics-tls\") pod \"dns-default-2859b\" (UID: \"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663\") " pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:31.464921 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.464848    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"image-registry-77fc874b44-g6g4r\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") " pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:31.627681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.627652    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nwnk6\""
Sep 11 16:19:31.627681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.627652    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:19:31.631995 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.631956    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:31.633837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.633821    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:31.633933 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.633899    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:31.698567 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.698480    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:31.918024 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.917950    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2859b"]
Sep 11 16:19:31.920736 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:31.920710    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de0b6f7_cfc7_4bb5_b33f_083ab3ce3663.slice/crio-4abc25b30701b436fac554198d4b8211f3e2e06500cbc325f984c0325853a34d WatchSource:0}: Error finding container 4abc25b30701b436fac554198d4b8211f3e2e06500cbc325f984c0325853a34d: Status 404 returned error can't find the container with id 4abc25b30701b436fac554198d4b8211f3e2e06500cbc325f984c0325853a34d
Sep 11 16:19:31.921655 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:31.921615    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:19:31.923442 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:19:31.923419    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2180806_0309_4f75_9196_5266cc1b3716.slice/crio-37cfbbf400e47780e46049ab2cb934506c4a4f1349f25fd34366bd5e5502356d WatchSource:0}: Error finding container 37cfbbf400e47780e46049ab2cb934506c4a4f1349f25fd34366bd5e5502356d: Status 404 returned error can't find the container with id 37cfbbf400e47780e46049ab2cb934506c4a4f1349f25fd34366bd5e5502356d
Sep 11 16:19:32.443323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:32.443291    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2859b" event={"ID":"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663","Type":"ContainerStarted","Data":"4abc25b30701b436fac554198d4b8211f3e2e06500cbc325f984c0325853a34d"}
Sep 11 16:19:32.444752 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:32.444727    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" event={"ID":"a2180806-0309-4f75-9196-5266cc1b3716","Type":"ContainerStarted","Data":"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"}
Sep 11 16:19:32.444867 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:32.444771    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" event={"ID":"a2180806-0309-4f75-9196-5266cc1b3716","Type":"ContainerStarted","Data":"37cfbbf400e47780e46049ab2cb934506c4a4f1349f25fd34366bd5e5502356d"}
Sep 11 16:19:32.444867 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:32.444855    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:32.469317 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:32.469268    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" podStartSLOduration=162.469253542 podStartE2EDuration="2m42.469253542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:32.467272521 +0000 UTC m=+163.241973099" watchObservedRunningTime="2026-09-11 16:19:32.469253542 +0000 UTC m=+163.243954119"
Sep 11 16:19:34.454174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:34.454136    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2859b" event={"ID":"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663","Type":"ContainerStarted","Data":"bcddd244e0cab697e47e8f17899d5524265c529c22d31c08de6a2ba234d69a91"}
Sep 11 16:19:34.454521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:34.454218    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2859b" event={"ID":"7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663","Type":"ContainerStarted","Data":"799e499b05daf4d056680a75ae1eaa3db19d067a6d7ef86b6081e5fff8039186"}
Sep 11 16:19:34.454521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:34.454258    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:34.475923 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:34.475873    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2859b" podStartSLOduration=129.42670933 podStartE2EDuration="2m11.47585992s" totalImagesPullingTime="2.04915059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:34.474907525 +0000 UTC m=+165.249608104" watchObservedRunningTime="2026-09-11 16:19:34.47585992 +0000 UTC m=+165.250560497"
Sep 11 16:19:36.588671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:36.588636    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:36.660574 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:36.660540    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:44.462785 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:44.462709    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2859b"
Sep 11 16:19:45.022471 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:45.022434    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:45.080126 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:45.080101    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:49.050233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:49.050197    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:49.154091 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:49.154062    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:50.716781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:50.716753    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:50.720532 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:50.720512    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5949df56d6-gvqpx"
Sep 11 16:19:50.761487 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:50.761458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:50.843935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:50.843904    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:51.638456 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:51.638426    2550 patch_prober.go:28] interesting pod/image-registry-77fc874b44-g6g4r container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:51.638656 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:51.638475    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" podUID="a2180806-0309-4f75-9196-5266cc1b3716" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:52.433342 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:52.433305    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:52.499061 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:52.499036    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:53.452309 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:53.452280    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:19:53.590874 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:53.590842    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:53.688220 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:53.688189    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:53.788956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:53.788868    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.036974 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:58.036936    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.093994 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:58.093961    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.006983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.006949    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.100321 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.100294    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.133932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.133907    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:19:59.241375 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.241341    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.383147 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.383067    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.480127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.480095    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.856655 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.856615    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.953407 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:19:59.953377    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:00.054065 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:00.054033    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:04.902481 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:04.902448    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:04.959755 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:04.959728    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.154069 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:06.154041    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.254933 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:06.254902    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.593371 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:06.593337    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.675543 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:06.675509    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:11.096978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:11.096935    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:12.621673 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:12.621641    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:12.640788 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:12.640764    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:13.619229 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:13.619203    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:13.903868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:13.903781    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:13.971475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:13.971442    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:21.590292 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:21.590254    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:21.662376 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:21.662348    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:22.212169 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:22.212136    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:22.274937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:22.274903    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:24.152678 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:24.152607    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" podUID="a2180806-0309-4f75-9196-5266cc1b3716" containerName="registry" containerID="cri-o://48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c" gracePeriod=30
Sep 11 16:20:25.395906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.395882    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:20:25.509204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509132    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509166    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509244    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509285    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509303    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509346    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509366    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509408    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6nkx\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx\") pod \"a2180806-0309-4f75-9196-5266cc1b3716\" (UID: \"a2180806-0309-4f75-9196-5266cc1b3716\") "
Sep 11 16:20:25.509968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509902    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:25.510060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.509977    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:25.511863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.511838    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:25.512018 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.511989    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:25.512102 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.512011    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:25.512142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.512094    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:25.512142 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.512134    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx" (OuterVolumeSpecName: "kube-api-access-z6nkx") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "kube-api-access-z6nkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:25.517859 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.517834    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a2180806-0309-4f75-9196-5266cc1b3716" (UID: "a2180806-0309-4f75-9196-5266cc1b3716"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:25.610157 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610134    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-registry-tls\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610157 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610155    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-image-registry-private-configuration\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610164    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2180806-0309-4f75-9196-5266cc1b3716-installation-pull-secrets\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610173    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-trusted-ca\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610180    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2180806-0309-4f75-9196-5266cc1b3716-ca-trust-extracted\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610188    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2180806-0309-4f75-9196-5266cc1b3716-registry-certificates\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610197    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-bound-sa-token\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.610270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.610206    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6nkx\" (UniqueName: \"kubernetes.io/projected/a2180806-0309-4f75-9196-5266cc1b3716-kube-api-access-z6nkx\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:25.650464 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.650438    2550 generic.go:356] "Generic (PLEG): container finished" podID="a2180806-0309-4f75-9196-5266cc1b3716" containerID="48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c" exitCode=0
Sep 11 16:20:25.650548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.650489    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r"
Sep 11 16:20:25.650548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.650518    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" event={"ID":"a2180806-0309-4f75-9196-5266cc1b3716","Type":"ContainerDied","Data":"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"}
Sep 11 16:20:25.650643 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.650564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77fc874b44-g6g4r" event={"ID":"a2180806-0309-4f75-9196-5266cc1b3716","Type":"ContainerDied","Data":"37cfbbf400e47780e46049ab2cb934506c4a4f1349f25fd34366bd5e5502356d"}
Sep 11 16:20:25.650643 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.650583    2550 scope.go:117] "RemoveContainer" containerID="48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"
Sep 11 16:20:25.660047 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.660028    2550 scope.go:117] "RemoveContainer" containerID="48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"
Sep 11 16:20:25.660340 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:25.660320    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c\": container with ID starting with 48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c not found: ID does not exist" containerID="48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"
Sep 11 16:20:25.660413 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.660347    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c"} err="failed to get container status \"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c\": rpc error: code = NotFound desc = could not find container \"48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c\": container with ID starting with 48288894a7d74af33a72975faf3afc9d2a072bb896843eceedb4910df838146c not found: ID does not exist"
Sep 11 16:20:25.672477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.672456    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:20:25.679024 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.679003    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77fc874b44-g6g4r"]
Sep 11 16:20:25.686045 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:25.686023    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2180806-0309-4f75-9196-5266cc1b3716" path="/var/lib/kubelet/pods/a2180806-0309-4f75-9196-5266cc1b3716/volumes"
Sep 11 16:20:26.522942 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.522911    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523174    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="alertmanager" containerID="cri-o://4ff01030cd97864e5bb28d51d6d0b32213b1739e852b4519c0779a3408c7b1e8" gracePeriod=120
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523202    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy" containerID="cri-o://b0601d48feb74a6db73be9dd2016918e4cd1a1d87a541a75840fb9659ee21544" gracePeriod=120
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523228    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-metric" containerID="cri-o://ca1fb9c2561dcc5f7347604b30075899a94a62a2779842d8c2056e93e794f06b" gracePeriod=120
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523239    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="config-reloader" containerID="cri-o://0d457063fb5072783c4fe3815eb17a01d82b79bb6a5482369ed2c0352b878db4" gracePeriod=120
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523220    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="prom-label-proxy" containerID="cri-o://a737f8c4bcae35333a2db6a2ef8710fa6ac8a785b549c85974d4a70bc41866b7" gracePeriod=120
Sep 11 16:20:26.523356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.523203    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-web" containerID="cri-o://2355465a1e746dddacaa9cfcafa8ee3c7234abddf8d4d6da7ac9eddd81bcf907" gracePeriod=120
Sep 11 16:20:26.661133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661097    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="a737f8c4bcae35333a2db6a2ef8710fa6ac8a785b549c85974d4a70bc41866b7" exitCode=0
Sep 11 16:20:26.661133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661129    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="ca1fb9c2561dcc5f7347604b30075899a94a62a2779842d8c2056e93e794f06b" exitCode=0
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661139    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="b0601d48feb74a6db73be9dd2016918e4cd1a1d87a541a75840fb9659ee21544" exitCode=0
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661149    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="2355465a1e746dddacaa9cfcafa8ee3c7234abddf8d4d6da7ac9eddd81bcf907" exitCode=0
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661160    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="0d457063fb5072783c4fe3815eb17a01d82b79bb6a5482369ed2c0352b878db4" exitCode=0
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661159    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"a737f8c4bcae35333a2db6a2ef8710fa6ac8a785b549c85974d4a70bc41866b7"}
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661226    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"ca1fb9c2561dcc5f7347604b30075899a94a62a2779842d8c2056e93e794f06b"}
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661168    2550 generic.go:356] "Generic (PLEG): container finished" podID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerID="4ff01030cd97864e5bb28d51d6d0b32213b1739e852b4519c0779a3408c7b1e8" exitCode=0
Sep 11 16:20:26.661294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"b0601d48feb74a6db73be9dd2016918e4cd1a1d87a541a75840fb9659ee21544"}
Sep 11 16:20:26.661513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661310    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"2355465a1e746dddacaa9cfcafa8ee3c7234abddf8d4d6da7ac9eddd81bcf907"}
Sep 11 16:20:26.661513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661335    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"0d457063fb5072783c4fe3815eb17a01d82b79bb6a5482369ed2c0352b878db4"}
Sep 11 16:20:26.661513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.661357    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"4ff01030cd97864e5bb28d51d6d0b32213b1739e852b4519c0779a3408c7b1e8"}
Sep 11 16:20:26.771983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.771960    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:26.919686 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919657    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919693    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919723    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919758    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919787    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919814    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.919857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919843    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919900    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919925    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919965    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.919992    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920030    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920071    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca\") pod \"1062c954-c2d5-4a49-9dab-f744b7c67215\" (UID: \"1062c954-c2d5-4a49-9dab-f744b7c67215\") "
Sep 11 16:20:26.920161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920130    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:26.920491 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920394    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-main-db\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:26.920551 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920484    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:26.920835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.920804    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:26.923158 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.923117    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.923615 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.923591    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8" (OuterVolumeSpecName: "kube-api-access-dzcl8") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "kube-api-access-dzcl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:26.923769 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.923749    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.923846 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.923769    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.924021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.923988    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.924123 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.924021    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:26.924235 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.924213    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume" (OuterVolumeSpecName: "config-volume") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.924373 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.924356    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out" (OuterVolumeSpecName: "config-out") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:26.926887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.926866    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:26.933534 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:26.933514    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config" (OuterVolumeSpecName: "web-config") pod "1062c954-c2d5-4a49-9dab-f744b7c67215" (UID: "1062c954-c2d5-4a49-9dab-f744b7c67215"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:27.021166 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021134    2550 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-cluster-tls-config\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021166 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021166    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021177    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021186    2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1062c954-c2d5-4a49-9dab-f744b7c67215-config-out\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021194    2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-tls-assets\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021202    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021223    2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-web-config\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021231    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1062c954-c2d5-4a49-9dab-f744b7c67215-metrics-client-ca\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021240    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/1062c954-c2d5-4a49-9dab-f744b7c67215-kube-api-access-dzcl8\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021248    2550 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-config-volume\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021255    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-main-tls\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.021300 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.021263    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1062c954-c2d5-4a49-9dab-f744b7c67215-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:27.135588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.135521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:27.200691 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.200665    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:27.669599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.669568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1062c954-c2d5-4a49-9dab-f744b7c67215","Type":"ContainerDied","Data":"3469a8927358dfab1549966fda91a1348c474a9f3d9841ac69daeb7b2cbf8839"}
Sep 11 16:20:27.669956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.669656    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.669956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.669687    2550 scope.go:117] "RemoveContainer" containerID="a737f8c4bcae35333a2db6a2ef8710fa6ac8a785b549c85974d4a70bc41866b7"
Sep 11 16:20:27.677647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.677618    2550 scope.go:117] "RemoveContainer" containerID="ca1fb9c2561dcc5f7347604b30075899a94a62a2779842d8c2056e93e794f06b"
Sep 11 16:20:27.684270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.684251    2550 scope.go:117] "RemoveContainer" containerID="b0601d48feb74a6db73be9dd2016918e4cd1a1d87a541a75840fb9659ee21544"
Sep 11 16:20:27.691616 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.691598    2550 scope.go:117] "RemoveContainer" containerID="2355465a1e746dddacaa9cfcafa8ee3c7234abddf8d4d6da7ac9eddd81bcf907"
Sep 11 16:20:27.694661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.694638    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:27.699922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.699893    2550 scope.go:117] "RemoveContainer" containerID="0d457063fb5072783c4fe3815eb17a01d82b79bb6a5482369ed2c0352b878db4"
Sep 11 16:20:27.700638 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.700600    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:27.706531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.706508    2550 scope.go:117] "RemoveContainer" containerID="4ff01030cd97864e5bb28d51d6d0b32213b1739e852b4519c0779a3408c7b1e8"
Sep 11 16:20:27.713031 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.713015    2550 scope.go:117] "RemoveContainer" containerID="a5f31398d3a73dc18fa31fe33b21d0bb1dbe8a193aa333b73e53ef9138bdef4f"
Sep 11 16:20:27.726820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.726799    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:27.727061 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727050    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727063    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727072    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="alertmanager"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727077    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="alertmanager"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727088    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-web"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727093    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-web"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727102    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="init-config-reloader"
Sep 11 16:20:27.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727107    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="init-config-reloader"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727113    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-metric"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727118    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-metric"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727125    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="prom-label-proxy"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727130    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="prom-label-proxy"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727142    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="config-reloader"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727147    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="config-reloader"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727153    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2180806-0309-4f75-9196-5266cc1b3716" containerName="registry"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727157    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2180806-0309-4f75-9196-5266cc1b3716" containerName="registry"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727201    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-web"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727210    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="config-reloader"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727215    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="prom-label-proxy"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727222    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="alertmanager"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727228    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2180806-0309-4f75-9196-5266cc1b3716" containerName="registry"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727234    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy"
Sep 11 16:20:27.727333 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.727240    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" containerName="kube-rbac-proxy-metric"
Sep 11 16:20:27.731068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.731052    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.734165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734146    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:20:27.734303 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734288    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:20:27.734594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734576    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9sj25\""
Sep 11 16:20:27.734751 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734594    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:20:27.734847 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734756    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:20:27.734902 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734603    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:20:27.734952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734638    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:20:27.734952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734641    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:20:27.735044 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.734596    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:20:27.739715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.739682    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:20:27.748074 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.748055    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:27.829640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829552    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829617    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829657    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829698    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbtk\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-kube-api-access-rtbtk\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829748    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829800    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829834    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.829935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829925    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.830193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.829950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931163    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931194    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931209    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931229    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931255    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931277    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbtk\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-kube-api-access-rtbtk\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931313    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931339    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931362    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931386    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931468    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.931900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.931609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.932120 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.932098    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.933062 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.933030    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2b9674-3a7a-46e8-a189-eef89f8defc7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934329    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934465 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934428    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934468    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934577 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934551    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934639    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.934831 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.934813    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.935105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.935082    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.935660 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.935609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.936416 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.936400    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b2b9674-3a7a-46e8-a189-eef89f8defc7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:27.941071 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:27.941050    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbtk\" (UniqueName: \"kubernetes.io/projected/8b2b9674-3a7a-46e8-a189-eef89f8defc7-kube-api-access-rtbtk\") pod \"alertmanager-main-0\" (UID: \"8b2b9674-3a7a-46e8-a189-eef89f8defc7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:28.040366 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.040340    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:20:28.316788 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.316731    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:20:28.318836 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:20:28.318807    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2b9674_3a7a_46e8_a189_eef89f8defc7.slice/crio-6cdefdd1677b2588c7bc5d6adbe92335c1480f9f75f58e46a925e7ade0373413 WatchSource:0}: Error finding container 6cdefdd1677b2588c7bc5d6adbe92335c1480f9f75f58e46a925e7ade0373413: Status 404 returned error can't find the container with id 6cdefdd1677b2588c7bc5d6adbe92335c1480f9f75f58e46a925e7ade0373413
Sep 11 16:20:28.675034 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.674955    2550 generic.go:356] "Generic (PLEG): container finished" podID="8b2b9674-3a7a-46e8-a189-eef89f8defc7" containerID="87c0e86433c138b16ab5d025e7d502402c995f7ea862f9a26d8cb65aa63a6983" exitCode=0
Sep 11 16:20:28.675410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.675035    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerDied","Data":"87c0e86433c138b16ab5d025e7d502402c995f7ea862f9a26d8cb65aa63a6983"}
Sep 11 16:20:28.675410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.675119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"6cdefdd1677b2588c7bc5d6adbe92335c1480f9f75f58e46a925e7ade0373413"}
Sep 11 16:20:28.803560 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.803531    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:28.903547 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:28.903518    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:29.687682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.687652    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1062c954-c2d5-4a49-9dab-f744b7c67215" path="/var/lib/kubelet/pods/1062c954-c2d5-4a49-9dab-f744b7c67215/volumes"
Sep 11 16:20:29.688455 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688428    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"553fc709e48bf3653b95a173739d15c56d2a65f0f5ebe10b71750c24c7e5e2e4"}
Sep 11 16:20:29.688552 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688517    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"caf92e7a02096db8f2c08d7de1a3c186c78d7e9d2f43e15bcd5a927926f9d387"}
Sep 11 16:20:29.688552 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688547    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"58eda929c6b1a68de51292e24c6ce53cc371f0b057a4ffc7ebc414e932d41066"}
Sep 11 16:20:29.688669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688586    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"c7b5e4e8d67dbbd0d894beeda252e7073952c94cacbcd0a88e93de5b910170c2"}
Sep 11 16:20:29.688669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688639    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"2ac11bc96adb5234128a8d53feabf26a188c64fa619e921e3617199bdcf7857f"}
Sep 11 16:20:29.688780 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.688677    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b2b9674-3a7a-46e8-a189-eef89f8defc7","Type":"ContainerStarted","Data":"eb39a619fa879e3fbce0edd11de9cb660be73dbbada360886a725cf997723d8f"}
Sep 11 16:20:29.715163 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:29.715115    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.698998406 podStartE2EDuration="2.715100281s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.101875ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:29.712359855 +0000 UTC m=+220.487060433" watchObservedRunningTime="2026-09-11 16:20:29.715100281 +0000 UTC m=+220.489800859"
Sep 11 16:20:30.232490 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.232458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:30.297227 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.297193    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:30.783394 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.783360    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:30.784447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.784344    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="prometheus" containerID="cri-o://306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2" gracePeriod=600
Sep 11 16:20:30.785450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.785064    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2" gracePeriod=600
Sep 11 16:20:30.785450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.785209    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy" containerID="cri-o://ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07" gracePeriod=600
Sep 11 16:20:30.785450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.785255    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="thanos-sidecar" containerID="cri-o://b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847" gracePeriod=600
Sep 11 16:20:30.785450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.785295    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-web" containerID="cri-o://8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102" gracePeriod=600
Sep 11 16:20:30.785450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:30.785335    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="config-reloader" containerID="cri-o://678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0" gracePeriod=600
Sep 11 16:20:31.020494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.020471    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.154920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.154843    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.154920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.154881    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.154920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.154902    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.154942    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.154974    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155026    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s8wn\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155063    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155088    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155110    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155135    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155177 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155169    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155196    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155265    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155289    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155300    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155313    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155348    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155385    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155418    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155410    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.155563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155444    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle\") pod \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\" (UID: \"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d\") "
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155584    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155822    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155841    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-metrics-client-ca\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.155857    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.156157    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.156353    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.157183    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:31.157477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.157315    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.157905 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.157838    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:31.158308 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.158244    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.158308 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.158270    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.158550 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.158351    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn" (OuterVolumeSpecName: "kube-api-access-8s8wn") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "kube-api-access-8s8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:31.158550 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.158399    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.158978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.158947    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out" (OuterVolumeSpecName: "config-out") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:31.159200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.159170    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.159492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.159473    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.159806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.159789    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.160082 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.160065    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.160251 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.160236    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config" (OuterVolumeSpecName: "config") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.160395 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.160377    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:31.168817 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.168795    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config" (OuterVolumeSpecName: "web-config") pod "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" (UID: "afe3cb50-ec96-4695-bdc6-0cfaef9eb63d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:31.256631 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256604    2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config-out\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256649    2550 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-metrics-client-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256663    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-db\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256676    2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-tls-assets\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256688    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256700    2550 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-config\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256714    2550 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.256738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256729    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256767    2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-web-config\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256780    2550 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-grpc-tls\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256793    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8wn\" (UniqueName: \"kubernetes.io/projected/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-kube-api-access-8s8wn\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256806    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256820    2550 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256834    2550 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-kube-rbac-proxy\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256848    2550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256864    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.257016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.256877    2550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:20:31.356599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.356414    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:31.455936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.455907    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:31.556919 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.556893    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:31.695947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695922    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2" exitCode=0
Sep 11 16:20:31.695947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695944    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07" exitCode=0
Sep 11 16:20:31.695947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695950    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102" exitCode=0
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695956    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847" exitCode=0
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695961    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0" exitCode=0
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.695967    2550 generic.go:356] "Generic (PLEG): container finished" podID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2" exitCode=0
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696004    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696002    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"}
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696062    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"}
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696089    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"}
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696113    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"}
Sep 11 16:20:31.696184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696136    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.696610 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696138    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"}
Sep 11 16:20:31.696610 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696230    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"}
Sep 11 16:20:31.696610 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.696267    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"afe3cb50-ec96-4695-bdc6-0cfaef9eb63d","Type":"ContainerDied","Data":"0a9038ac452d65c4229013e040d1cb26e1a67807692a7094aef427de9208d08b"}
Sep 11 16:20:31.703766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.703751    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.710482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.710468    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.717165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.717146    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.719374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.719354    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:31.724281 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.724258    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:31.725161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.725145    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.731548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.731531    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.738269 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.738253    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.744837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.744821    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.745112 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.745094    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.745164 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745120    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.745164 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745138    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.745400 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.745382    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.745440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745406    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.745440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745423    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.745655 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.745638    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.745714 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745663    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.745714 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745683    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.745918 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.745901    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.745956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745922    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.745956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.745934    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.746142 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.746125    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.746197 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746148    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.746197 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746168    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.746378 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.746363    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.746417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746380    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.746417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746392    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.746632 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:20:31.746605    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.746689 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746639    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.746689 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746658    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.746905 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746880    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.746905 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.746905    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.747136 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747109    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.747206 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747137    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.747323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747304    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.747393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747325    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.747585 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747564    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.747585 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747583    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.747837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747818    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.747906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.747838    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.748089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748071    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.748139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748089    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.748311 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748294    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.748372 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748312    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.748539 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748522    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.748582 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748539    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.748796 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748775    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.748854 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.748805    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.749049 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749033    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.749109 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749050    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.749295 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749275    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.749358 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749297    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.749511 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749494    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.749575 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749512    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.749765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749748    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.749809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749765    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.749964 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749948    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.750010 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.749964    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.750163 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750146    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.750227 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750165    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.750366 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750351    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.750411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750366    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.750557 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750543    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.750602 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750557    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.750766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750751    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.750813 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750766    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.750968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750954    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.751004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.750969    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.751184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751166    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.751260 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751185    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.751456 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751433    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.751456 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751456    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.751917 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751890    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.751917 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.751918    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.752353 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752328    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.752353 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752353    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.752657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752610    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.752657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752652    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.752913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752892    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.752913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.752913    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.753127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753111    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.753184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753133    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.753372 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753352    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.753429 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753374    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.753580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753565    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.753655 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753579    2550 scope.go:117] "RemoveContainer" containerID="b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"
Sep 11 16:20:31.753829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753811    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:31.753887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753823    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2"} err="failed to get container status \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": rpc error: code = NotFound desc = could not find container \"b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2\": container with ID starting with b37ea5fae553fbced91435e01d1eca83b96712fe3cd3858e4b992f82c5e5c9f2 not found: ID does not exist"
Sep 11 16:20:31.753887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.753848    2550 scope.go:117] "RemoveContainer" containerID="ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"
Sep 11 16:20:31.754095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754067    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07"} err="failed to get container status \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": rpc error: code = NotFound desc = could not find container \"ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07\": container with ID starting with ab7ea6f797a0f27b418bb921221b7a5414d3f070e50765ed0e65337845c20a07 not found: ID does not exist"
Sep 11 16:20:31.754095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754094    2550 scope.go:117] "RemoveContainer" containerID="8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"
Sep 11 16:20:31.754204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754174    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-web"
Sep 11 16:20:31.754204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754190    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-web"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754211    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754220    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754229    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754237    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754254    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="init-config-reloader"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754262    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="init-config-reloader"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754276    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="thanos-sidecar"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754279    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102"} err="failed to get container status \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": rpc error: code = NotFound desc = could not find container \"8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102\": container with ID starting with 8ac15eef8ed447ea0719db31cef19978b6b84e84b16e719fa690df04b4065102 not found: ID does not exist"
Sep 11 16:20:31.754312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754299    2550 scope.go:117] "RemoveContainer" containerID="b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754285    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="thanos-sidecar"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754349    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="prometheus"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754356    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="prometheus"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754365    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="config-reloader"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754370    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="config-reloader"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754440    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-web"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754454    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754464    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="config-reloader"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754471    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="thanos-sidecar"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754479    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="kube-rbac-proxy"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754488    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" containerName="prometheus"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754520    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847"} err="failed to get container status \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": rpc error: code = NotFound desc = could not find container \"b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847\": container with ID starting with b9b76c0bc0adb0199815d4ab122d185ee8876806dad973fff49ca79b24ce7847 not found: ID does not exist"
Sep 11 16:20:31.754603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754543    2550 scope.go:117] "RemoveContainer" containerID="678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"
Sep 11 16:20:31.755039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754815    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0"} err="failed to get container status \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": rpc error: code = NotFound desc = could not find container \"678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0\": container with ID starting with 678c1ed4fae4f3e38d3acb1c1c40fe53e60dd0747eee0ae142214ee8cdf3c6a0 not found: ID does not exist"
Sep 11 16:20:31.755039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.754833    2550 scope.go:117] "RemoveContainer" containerID="306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"
Sep 11 16:20:31.755143 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.755073    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2"} err="failed to get container status \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": rpc error: code = NotFound desc = could not find container \"306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2\": container with ID starting with 306e8a65a8c1dc279443d578b09fa73ff6121f55e2cb71b96a4e4d2a6329d8e2 not found: ID does not exist"
Sep 11 16:20:31.755143 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.755096    2550 scope.go:117] "RemoveContainer" containerID="6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"
Sep 11 16:20:31.755343 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.755325    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e"} err="failed to get container status \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": rpc error: code = NotFound desc = could not find container \"6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e\": container with ID starting with 6e324ed1b5e8f842c27bf9c111ffda17368326f24f78a07deb956f4b3935211e not found: ID does not exist"
Sep 11 16:20:31.758545 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.758529    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.761030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761011    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:20:31.761174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761130    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:20:31.761174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761169    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:20:31.761493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761478    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:20:31.761578 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761496    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:20:31.761578 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761481    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:20:31.761578 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761532    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:20:31.761983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.761968    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:20:31.762073 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762024    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:20:31.762128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762097    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7g5df\""
Sep 11 16:20:31.762179 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762166    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:20:31.762230 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762215    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6k6vvudrnq33p\""
Sep 11 16:20:31.762492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762478    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:20:31.762559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.762479    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:20:31.764849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.764828    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:20:31.768022 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.768004    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:20:31.776534 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.776505    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:31.861579 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861554    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861583    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861662    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861688    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861719    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861790    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.861935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861917    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-config-out\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.861986    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862006    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-web-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862186    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862217    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862527 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862507    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862576 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862546    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862618 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w824f\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-kube-api-access-w824f\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862618 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862612    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.862700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.862672    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963356    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-config-out\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963382    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963403    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963420    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963454    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-web-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963519    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963564    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963585    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w824f\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-kube-api-access-w824f\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963609    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963682    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963758    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963783    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.963830 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963806    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963840    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.963896    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.964126    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964233 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.964158    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964883 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.964856    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.964968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.964898    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.965428 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.965403    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.966558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.966531    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37f7e6ae-9bef-4137-96d2-a36198da4d39-config-out\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.966659 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.966554    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.966659 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.966614    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.966900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.966876    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.967747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.967203    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.967747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.967402    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.967747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.967681    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-web-config\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.967943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.967837    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.968093 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.968068    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.969092 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.969066    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.969189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.969096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.969255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.969239    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.969608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.969589    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/37f7e6ae-9bef-4137-96d2-a36198da4d39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.969994 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.969977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/37f7e6ae-9bef-4137-96d2-a36198da4d39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:31.971726 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:31.971709    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w824f\" (UniqueName: \"kubernetes.io/projected/37f7e6ae-9bef-4137-96d2-a36198da4d39-kube-api-access-w824f\") pod \"prometheus-k8s-0\" (UID: \"37f7e6ae-9bef-4137-96d2-a36198da4d39\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:32.068192 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:32.068165    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:32.345378 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:32.345351    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:32.347543 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:20:32.347509    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f7e6ae_9bef_4137_96d2_a36198da4d39.slice/crio-f6fdf36b027080731298272660e77e50edc1f043954398fa8e3e89c9d106b23e WatchSource:0}: Error finding container f6fdf36b027080731298272660e77e50edc1f043954398fa8e3e89c9d106b23e: Status 404 returned error can't find the container with id f6fdf36b027080731298272660e77e50edc1f043954398fa8e3e89c9d106b23e
Sep 11 16:20:32.700987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:32.700955    2550 generic.go:356] "Generic (PLEG): container finished" podID="37f7e6ae-9bef-4137-96d2-a36198da4d39" containerID="c3245b3ce63fff5beb9d9fba470e0d235f4ff3c9e1115aef4765d8b01014663e" exitCode=0
Sep 11 16:20:32.701139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:32.701048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerDied","Data":"c3245b3ce63fff5beb9d9fba470e0d235f4ff3c9e1115aef4765d8b01014663e"}
Sep 11 16:20:32.701139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:32.701107    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"f6fdf36b027080731298272660e77e50edc1f043954398fa8e3e89c9d106b23e"}
Sep 11 16:20:33.686636 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.686593    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe3cb50-ec96-4695-bdc6-0cfaef9eb63d" path="/var/lib/kubelet/pods/afe3cb50-ec96-4695-bdc6-0cfaef9eb63d/volumes"
Sep 11 16:20:33.708818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708792    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"2357aeed03e7635ba898a1cd4bad28afc633ca05bec741d979b7a8e110aa0351"}
Sep 11 16:20:33.708932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708844    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"79ac805c0fe9146277aaae5e2731da3713e4e2a02979fdd1d7a07baed0b7779c"}
Sep 11 16:20:33.708932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708871    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"ba8ab7e2b449382944bc598107feddac6b1bd4b4b6d59e8cc8dea6917fb683f8"}
Sep 11 16:20:33.708932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708895    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"1408e0814a38125833b1b066cb6fb30db8bcc7261b6eb062db38d9fcf2065d62"}
Sep 11 16:20:33.708932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708919    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"9a04e42bfbec672f103800e5203ec92439ba41861c55e194cd7200a980558075"}
Sep 11 16:20:33.709067 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.708944    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"37f7e6ae-9bef-4137-96d2-a36198da4d39","Type":"ContainerStarted","Data":"02e58a2f78067037eb215476d915139649a8ccd03c1ac5b5d9af4a7fc55014d9"}
Sep 11 16:20:33.740351 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:33.740306    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7198427 podStartE2EDuration="2.740293015s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.450315ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:33.73746447 +0000 UTC m=+224.512165041" watchObservedRunningTime="2026-09-11 16:20:33.740293015 +0000 UTC m=+224.514993592"
Sep 11 16:20:36.599079 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:36.599043    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:36.665412 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:36.665386    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:37.068686 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:37.068656    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:40.150434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:40.150399    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:40.249900 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:40.249869    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:40.306396 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:20:40.306371    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:32.069366 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:21:32.069325    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:32.084635 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:21:32.084603    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:32.946448 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:21:32.946424    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:49.586165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:21:49.586139    2550 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:22:21.021288 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:22:21.021253    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:39.554939 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:22:39.554900    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:14.448818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:23:14.448777    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:15.731811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:23:15.731783    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:36.604264 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:23:36.604234    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:36.706839 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:23:36.706807    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:36.986414 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:23:36.986386    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:37.328645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:24:37.328583    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:37.427916 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:24:37.427884    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:37.493278 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:24:37.493246    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:04.881022 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:04.880986    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:04.975148 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:04.975111    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:07.033316 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:07.033289    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:07.102687 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:07.102661    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:07.171083 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:07.171058    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:07.762944 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:07.762912    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:07.862670 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:07.862642    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:10.526003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:10.525962    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.109574 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.109544    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.209163 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.209128    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.430743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.430659    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.684084 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.683984    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.798360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.798334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.897975 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.897948    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:11.999145 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:11.999125    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:12.094734 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:12.094712    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:12.195169 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:12.195144    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:12.295519 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:12.295459    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:12.356946 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:12.356922    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:15.358010 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:15.357979    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:18.227663 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:18.227614    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:48.016857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:48.016763    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:48.076095 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:25:48.076070    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:03.070212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:03.070181    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:03.171613 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:03.171568    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:14.221460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:14.221419    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:14.976587 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:14.976548    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:18.030389 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:18.030355    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:18.131120 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:18.131066    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:20.619302 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:20.619267    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:20.719274 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:20.719244    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:22.672654 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.672611    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"]
Sep 11 16:26:22.673167 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.673040    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:22.675984 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.675970    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.678813 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.678787    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 16:26:22.680009 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.679993    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-b2pcr\""
Sep 11 16:26:22.680076 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.680048    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 16:26:22.683984 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.683964    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"]
Sep 11 16:26:22.772611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.772587    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.772757 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.772704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4tm\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-kube-api-access-dv4tm\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.818978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.818955    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:22.873506 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.873480    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4tm\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-kube-api-access-dv4tm\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.873610 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.873521    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.874499 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.874478    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:22.881436 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.881407    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.881651 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.881616    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4tm\" (UniqueName: \"kubernetes.io/projected/b95c7b29-4273-406b-b9e5-b2685e8e60ac-kube-api-access-dv4tm\") pod \"cert-manager-cainjector-559cdd8c87-m8dj5\" (UID: \"b95c7b29-4273-406b-b9e5-b2685e8e60ac\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:22.996877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:22.996857    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"
Sep 11 16:26:23.011147 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:23.011122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:23.265557 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:23.265534    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5"]
Sep 11 16:26:23.268069 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:26:23.268043    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb95c7b29_4273_406b_b9e5_b2685e8e60ac.slice/crio-c3320d57035a5e33a232606b5b1cb485c6f57b81528843cf3fa48b071c448817 WatchSource:0}: Error finding container c3320d57035a5e33a232606b5b1cb485c6f57b81528843cf3fa48b071c448817: Status 404 returned error can't find the container with id c3320d57035a5e33a232606b5b1cb485c6f57b81528843cf3fa48b071c448817
Sep 11 16:26:23.270293 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:23.270276    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:26:24.025324 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:24.025286    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5" event={"ID":"b95c7b29-4273-406b-b9e5-b2685e8e60ac","Type":"ContainerStarted","Data":"c3320d57035a5e33a232606b5b1cb485c6f57b81528843cf3fa48b071c448817"}
Sep 11 16:26:24.448374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:24.448343    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:24.551219 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:24.551177    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:24.650760 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:24.650700    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:25.102794 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:25.102759    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:25.155507 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:25.155464    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:27.041092 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.041055    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5" event={"ID":"b95c7b29-4273-406b-b9e5-b2685e8e60ac","Type":"ContainerStarted","Data":"180a2f67de8191e1060c142fae1578bd977afe6027c57c5d669deb3f41e23a3d"}
Sep 11 16:26:27.057154 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.057110    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m8dj5" podStartSLOduration=1.765072216 podStartE2EDuration="5.057096364s" totalImagesPullingTime="3.292024148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:27.056087464 +0000 UTC m=+577.830788053" watchObservedRunningTime="2026-09-11 16:26:27.057096364 +0000 UTC m=+577.831796941"
Sep 11 16:26:27.400669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.400638    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvc9"]
Sep 11 16:26:27.403442 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.403427    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.406067 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.406046    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8bf8c\""
Sep 11 16:26:27.412238 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.412214    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvc9"]
Sep 11 16:26:27.415213 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.415191    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.415331 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.415244    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nw7\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-kube-api-access-l8nw7\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.516285 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.516256    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nw7\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-kube-api-access-l8nw7\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.516429 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.516325    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.524484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.524453    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.524616 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.524528    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nw7\" (UniqueName: \"kubernetes.io/projected/4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7-kube-api-access-l8nw7\") pod \"cert-manager-webhook-6486ffd997-2nvc9\" (UID: \"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.712451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.712432    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:27.986933 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:27.986912    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvc9"]
Sep 11 16:26:27.989259 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:26:27.989231    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d9bd453_0ae0_4bfa_8bf2_f368fe3d97e7.slice/crio-fb8e8cfec55f47c300a82ccdd7a9413e2ae97a958ba61b60afc372e115cceb1b WatchSource:0}: Error finding container fb8e8cfec55f47c300a82ccdd7a9413e2ae97a958ba61b60afc372e115cceb1b: Status 404 returned error can't find the container with id fb8e8cfec55f47c300a82ccdd7a9413e2ae97a958ba61b60afc372e115cceb1b
Sep 11 16:26:28.046844 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:28.046818    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9" event={"ID":"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7","Type":"ContainerStarted","Data":"fb8e8cfec55f47c300a82ccdd7a9413e2ae97a958ba61b60afc372e115cceb1b"}
Sep 11 16:26:28.236775 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:28.236699    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:28.294524 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:28.294489    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:29.052669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:29.052637    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9" event={"ID":"4d9bd453-0ae0-4bfa-8bf2-f368fe3d97e7","Type":"ContainerStarted","Data":"9d8da73be61a1af90e86f0c8fa3a8a6bbb94546b776f534c6a5aeac75540a6e8"}
Sep 11 16:26:29.053007 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:29.052755    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:29.071291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:29.071249    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9" podStartSLOduration=2.071236423 podStartE2EDuration="2.071236423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:29.070088955 +0000 UTC m=+579.844789532" watchObservedRunningTime="2026-09-11 16:26:29.071236423 +0000 UTC m=+579.845937000"
Sep 11 16:26:31.422835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:31.422794    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:35.058937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:35.058907    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvc9"
Sep 11 16:26:36.419719 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:36.419690    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:36.520275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:36.520247    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:48.218387 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:48.218348    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:52.512511 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.512477    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:52.580228 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.580203    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"]
Sep 11 16:26:52.582982 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.582965    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.587013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.586986    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:26:52.587013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.587005    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lk58v\""
Sep 11 16:26:52.587204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.587013    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 16:26:52.587204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.587045    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 16:26:52.587204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.587089    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 16:26:52.587204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.587161    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 16:26:52.592741 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.592721    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"]
Sep 11 16:26:52.599505 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.599486    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-metrics-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.599594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.599512    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.599594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.599543    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2f8\" (UniqueName: \"kubernetes.io/projected/2e39d478-1232-4dcf-9553-61b776e9708c-kube-api-access-9c2f8\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.599594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.599582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e39d478-1232-4dcf-9553-61b776e9708c-manager-config\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.610531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.610508    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:52.700382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.700354    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-metrics-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.700531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.700394    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.700531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.700440    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2f8\" (UniqueName: \"kubernetes.io/projected/2e39d478-1232-4dcf-9553-61b776e9708c-kube-api-access-9c2f8\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.700930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.700906    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e39d478-1232-4dcf-9553-61b776e9708c-manager-config\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.701613 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.701588    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2e39d478-1232-4dcf-9553-61b776e9708c-manager-config\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.702941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.702916    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-metrics-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.703015 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.702960    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e39d478-1232-4dcf-9553-61b776e9708c-cert\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.709796 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.709775    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2f8\" (UniqueName: \"kubernetes.io/projected/2e39d478-1232-4dcf-9553-61b776e9708c-kube-api-access-9c2f8\") pod \"lws-controller-manager-5956778578-6jkf6\" (UID: \"2e39d478-1232-4dcf-9553-61b776e9708c\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:52.892848 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:52.892769    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:53.174600 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:53.174576    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"]
Sep 11 16:26:53.177333 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:26:53.177296    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e39d478_1232_4dcf_9553_61b776e9708c.slice/crio-6f625c1667334040dcf174315116f334f2465782600b85b9db071cf73f26a01f WatchSource:0}: Error finding container 6f625c1667334040dcf174315116f334f2465782600b85b9db071cf73f26a01f: Status 404 returned error can't find the container with id 6f625c1667334040dcf174315116f334f2465782600b85b9db071cf73f26a01f
Sep 11 16:26:53.623679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:53.623643    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:54.164109 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:54.164059    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6" event={"ID":"2e39d478-1232-4dcf-9553-61b776e9708c","Type":"ContainerStarted","Data":"6f625c1667334040dcf174315116f334f2465782600b85b9db071cf73f26a01f"}
Sep 11 16:26:56.175519 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:56.175478    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6" event={"ID":"2e39d478-1232-4dcf-9553-61b776e9708c","Type":"ContainerStarted","Data":"1e9eed995baf12671240cf4e99b26234af10193f48d683a4f8b0ce4a82687357"}
Sep 11 16:26:56.175982 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:56.175589    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:26:56.196804 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:26:56.196765    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6" podStartSLOduration=1.8700445989999999 podStartE2EDuration="4.196751796s" totalImagesPullingTime="2.326707197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:56.195263207 +0000 UTC m=+606.969963785" watchObservedRunningTime="2026-09-11 16:26:56.196751796 +0000 UTC m=+606.971452373"
Sep 11 16:27:00.760758 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:00.760725    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:00.862015 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:00.861986    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.129787 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.129754    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"]
Sep 11 16:27:02.135915 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.135891    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.139277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.139260    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:27:02.139277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.139269    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 16:27:02.139446 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.139313    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 16:27:02.139446 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.139407    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-qqkrr\""
Sep 11 16:27:02.150868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.150844    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"]
Sep 11 16:27:02.173787 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.173754    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.175880 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.175861    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.175957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.175889    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.175957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.175917    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.175957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.175939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.176116 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.176012    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.176116 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.176078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.176116 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.176110    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.176234 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.176137    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjw5\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-kube-api-access-kbjw5\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.176234 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.176183    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.244380 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.244344    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.276542 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276515    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276549    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276571    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276591    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276665    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276740    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276774    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjw5\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-kube-api-access-kbjw5\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.276863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.277043 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.276991    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.277043 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.277026    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-data\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.277146 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.277070    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-credential-socket\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.277387 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.277363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-workload-certs\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.277495 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.277461    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.278996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.278977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.279288 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.279271    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.284968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.284942    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-istio-token\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.285127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.285111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjw5\" (UniqueName: \"kubernetes.io/projected/83a68164-af95-4ca7-8e2c-1a55a7e2ce40-kube-api-access-kbjw5\") pod \"openshift-ai-inference-openshift-default-68dcb6f84c-svpft\" (UID: \"83a68164-af95-4ca7-8e2c-1a55a7e2ce40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.446854 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.446833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:02.719703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:02.719679    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"]
Sep 11 16:27:02.721759 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:27:02.721729    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a68164_af95_4ca7_8e2c_1a55a7e2ce40.slice/crio-5171621ef97aedce9d7f196f89b95f8ba51dd2a06042c8f8cd3781a6cec03c9b WatchSource:0}: Error finding container 5171621ef97aedce9d7f196f89b95f8ba51dd2a06042c8f8cd3781a6cec03c9b: Status 404 returned error can't find the container with id 5171621ef97aedce9d7f196f89b95f8ba51dd2a06042c8f8cd3781a6cec03c9b
Sep 11 16:27:03.202337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:03.202306    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" event={"ID":"83a68164-af95-4ca7-8e2c-1a55a7e2ce40","Type":"ContainerStarted","Data":"5171621ef97aedce9d7f196f89b95f8ba51dd2a06042c8f8cd3781a6cec03c9b"}
Sep 11 16:27:05.721287 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:05.721240    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:05.869509 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:05.869450    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:27:05.869596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:05.869550    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:27:05.869596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:05.869580    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:27:06.219396 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:06.219361    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" event={"ID":"83a68164-af95-4ca7-8e2c-1a55a7e2ce40","Type":"ContainerStarted","Data":"c4f37d9f1dd462906f0f10917d79a03661623e7e9469ed684b0eca72d0c5a6c0"}
Sep 11 16:27:06.245700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:06.245650    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podStartSLOduration=1.100277518 podStartE2EDuration="4.245633706s" totalImagesPullingTime="3.145356188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:06.243975913 +0000 UTC m=+617.018676483" watchObservedRunningTime="2026-09-11 16:27:06.245633706 +0000 UTC m=+617.020334277"
Sep 11 16:27:06.447779 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:06.447746    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:06.449063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:06.449040    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:27:06.449179 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:06.449089    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:27:07.181541 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:07.181511    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5956778578-6jkf6"
Sep 11 16:27:07.451161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:07.451135    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:08.229052 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:08.229024    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:08.230014 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:08.229989    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:27:08.418325 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:08.418298    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:08.529284 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:08.529210    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:23.127980 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:23.127944    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:25.938059 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:25.938022    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:26.707082 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:26.707052    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:27.936173 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:27.936145    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:28.166291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:28.166260    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:28.535511 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:28.535484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:28.630306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:28.630275    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:30.183800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:30.183764    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:31.595851 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:31.595815    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:31.736853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:31.736819    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.322365 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:32.322334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.585066 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:32.584986    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.678088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:32.678055    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.781796 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:32.781771    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.891034 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:32.890949    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:35.495404 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.493057    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"]
Sep 11 16:27:35.501892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.501863    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.504521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.504491    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:27:35.504521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.504512    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:27:35.504709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.504545    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xkmfd\""
Sep 11 16:27:35.510017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.509986    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"]
Sep 11 16:27:35.653501 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.653469    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4060dd60-2fee-4751-9a71-0397b53311c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.653679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.653527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pk6p\" (UniqueName: \"kubernetes.io/projected/4060dd60-2fee-4751-9a71-0397b53311c2-kube-api-access-4pk6p\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.753929 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.753868    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pk6p\" (UniqueName: \"kubernetes.io/projected/4060dd60-2fee-4751-9a71-0397b53311c2-kube-api-access-4pk6p\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.754039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.753935    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4060dd60-2fee-4751-9a71-0397b53311c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.754220 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.754205    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4060dd60-2fee-4751-9a71-0397b53311c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.762002 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.761977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pk6p\" (UniqueName: \"kubernetes.io/projected/4060dd60-2fee-4751-9a71-0397b53311c2-kube-api-access-4pk6p\") pod \"kuadrant-operator-controller-manager-6d4cb79c-dww5t\" (UID: \"4060dd60-2fee-4751-9a71-0397b53311c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:35.812699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:35.812670    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:36.098511 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:36.098488    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"]
Sep 11 16:27:36.101130 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:27:36.101094    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4060dd60_2fee_4751_9a71_0397b53311c2.slice/crio-824e624e97e0728404e3cd4aa4a5f645e5a82dfc44ec68b0aba8065f251cef46 WatchSource:0}: Error finding container 824e624e97e0728404e3cd4aa4a5f645e5a82dfc44ec68b0aba8065f251cef46: Status 404 returned error can't find the container with id 824e624e97e0728404e3cd4aa4a5f645e5a82dfc44ec68b0aba8065f251cef46
Sep 11 16:27:36.347538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:36.347505    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t" event={"ID":"4060dd60-2fee-4751-9a71-0397b53311c2","Type":"ContainerStarted","Data":"824e624e97e0728404e3cd4aa4a5f645e5a82dfc44ec68b0aba8065f251cef46"}
Sep 11 16:27:37.936606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:37.936567    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:38.043609 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:38.043531    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:39.748874 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:39.748843    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:39.845976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:39.845950    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:40.369153 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:40.369121    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t" event={"ID":"4060dd60-2fee-4751-9a71-0397b53311c2","Type":"ContainerStarted","Data":"a1fba56b6b96d72f5e5d4cfa6e75bb1116a3a41674ca6ba1a3636b3e8adc8f69"}
Sep 11 16:27:40.369315 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:40.369210    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:40.390133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:40.390086    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t" podStartSLOduration=1.597752633 podStartE2EDuration="5.390073472s" totalImagesPullingTime="3.792320839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:40.389529978 +0000 UTC m=+651.164230558" watchObservedRunningTime="2026-09-11 16:27:40.390073472 +0000 UTC m=+651.164774051"
Sep 11 16:27:42.366293 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:42.366250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:43.128373 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:43.128344    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:45.744408 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:45.744373    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:45.845562 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:45.845538    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:50.496185 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:50.496141    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:50.595849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:50.595814    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:50.662329 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:50.662296    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:51.375330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:51.375299    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-dww5t"
Sep 11 16:27:57.129185 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:57.129150    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:57.231973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:27:57.231943    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:00.128434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:00.128403    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:00.230399 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:00.230369    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:01.669064 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:01.669031    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.913855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:05.913824    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:06.014358 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:06.014326    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:06.522753 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:06.522724    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:06.624384 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:06.624353    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:09.985251 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:09.985219    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:10.084639 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:10.084596    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:10.185186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:10.185164    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:13.172445 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:13.172413    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:26.050911 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.050877    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:26.465395 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.465365    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"]
Sep 11 16:28:26.467634 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.467602    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.470250 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.470224    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 16:28:26.470348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.470224    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2r47n\""
Sep 11 16:28:26.476337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.476312    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"]
Sep 11 16:28:26.505148 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.502480    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"]
Sep 11 16:28:26.569665 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.569612    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-config-file\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.569812 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.569684    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55md\" (UniqueName: \"kubernetes.io/projected/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-kube-api-access-f55md\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.671094 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.671000    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-config-file\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.671235 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.671131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55md\" (UniqueName: \"kubernetes.io/projected/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-kube-api-access-f55md\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.671559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.671541    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-config-file\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.680439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.680410    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55md\" (UniqueName: \"kubernetes.io/projected/e0d771fc-bbb3-4ae9-88de-4f3b5a731810-kube-api-access-f55md\") pod \"limitador-limitador-84994f7ffc-8vgwp\" (UID: \"e0d771fc-bbb3-4ae9-88de-4f3b5a731810\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:26.778911 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:26.778849    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:27.047733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:27.047712    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"]
Sep 11 16:28:27.049970 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:28:27.049940    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d771fc_bbb3_4ae9_88de_4f3b5a731810.slice/crio-9c0a1ee36fef224450c629c7a31f8097971511647ef22e0843d7557a06a508b3 WatchSource:0}: Error finding container 9c0a1ee36fef224450c629c7a31f8097971511647ef22e0843d7557a06a508b3: Status 404 returned error can't find the container with id 9c0a1ee36fef224450c629c7a31f8097971511647ef22e0843d7557a06a508b3
Sep 11 16:28:27.425968 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:27.425891    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:27.570359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:27.570326    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp" event={"ID":"e0d771fc-bbb3-4ae9-88de-4f3b5a731810","Type":"ContainerStarted","Data":"9c0a1ee36fef224450c629c7a31f8097971511647ef22e0843d7557a06a508b3"}
Sep 11 16:28:28.519826 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.519791    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:28.575664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.575609    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp" event={"ID":"e0d771fc-bbb3-4ae9-88de-4f3b5a731810","Type":"ContainerStarted","Data":"3159cb414413358edfea1ba4933e744970742d6f6a849c597b54211a2dab75a6"}
Sep 11 16:28:28.575813 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.575764    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:28.594680 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.594636    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp" podStartSLOduration=1.702210489 podStartE2EDuration="2.594604714s" totalImagesPullingTime="892.394225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:28.593215243 +0000 UTC m=+699.367915823" watchObservedRunningTime="2026-09-11 16:28:28.594604714 +0000 UTC m=+699.369305291"
Sep 11 16:28:28.598419 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.598391    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:28.857803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:28.857721    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:39.583682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:39.583650    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8vgwp"
Sep 11 16:28:45.585667 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:45.585557    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:45.687174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:45.687144    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:53.781759 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:53.781728    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:53.880719 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:28:53.880686    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:04.463380 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:04.463345    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:04.552394 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:04.552365    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:13.890715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:13.890683    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:13.990903 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:13.990874    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.032406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.032379    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.112651 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.112608    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.215674 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.215640    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.335925 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.335839    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.436191 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.436167    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.643395 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.643315    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.714420 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.714387    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:19.776471 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:29:19.776443    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:09.637744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:09.637709    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:36.825159 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:36.825125    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:36.907523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:36.907488    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:37.008037 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:37.008006    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:37.112496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:37.112428    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:37.404715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:37.404635    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:37.856736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:37.856705    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:37.957159 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:37.957132    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:53.542870 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.542834    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kt8qc"]
Sep 11 16:30:53.546155 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.546139    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.548806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.548779    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:30:53.548806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.548798    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:30:53.549073 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.548946    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k9t7c\""
Sep 11 16:30:53.549991 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.549969    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:30:53.550573 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.550016    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:30:53.553282 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.553076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kt8qc"]
Sep 11 16:30:53.577440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.577413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.577537 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.577447    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffhg\" (UniqueName: \"kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.678783 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.678757    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.678883 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.678788    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffhg\" (UniqueName: \"kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.679096 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.679078    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.687003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.686976    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffhg\" (UniqueName: \"kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg\") pod \"s3-init-kt8qc\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") " pod="kserve/s3-init-kt8qc"
Sep 11 16:30:53.855538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:53.855482    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kt8qc"
Sep 11 16:30:54.124244 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:54.124221    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kt8qc"]
Sep 11 16:30:54.127026 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:30:54.126995    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33c4a3f0_0ae4_4dbc_8037_ead1f08c9f3f.slice/crio-4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89 WatchSource:0}: Error finding container 4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89: Status 404 returned error can't find the container with id 4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89
Sep 11 16:30:54.213771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:54.213731    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerStarted","Data":"4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89"}
Sep 11 16:30:58.238679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:30:58.238637    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerStarted","Data":"4d87d7a15e1269849263dfd5de96ab5d4d9c784c9b2cd348f7dd6e81e8d44bcc"}
Sep 11 16:31:00.250821 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:31:00.250784    2550 generic.go:356] "Generic (PLEG): container finished" podID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerID="4d87d7a15e1269849263dfd5de96ab5d4d9c784c9b2cd348f7dd6e81e8d44bcc" exitCode=0
Sep 11 16:31:00.251230 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:31:00.250855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerDied","Data":"4d87d7a15e1269849263dfd5de96ab5d4d9c784c9b2cd348f7dd6e81e8d44bcc"}
Sep 11 16:31:05.278469 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:31:05.278437    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerStarted","Data":"037a0e390fbb49919c273021e87f611f7d8de83b83292992a90dff54bbff1a85"}
Sep 11 16:31:05.296817 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:31:05.296764    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kt8qc" podStartSLOduration=2.23430258 podStartE2EDuration="12.296745355s" totalImagesPullingTime="8.314990369s" totalInitContainerRuntime="1.747452406s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:30:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:05.294825045 +0000 UTC m=+856.069525624" watchObservedRunningTime="2026-09-11 16:31:05.296745355 +0000 UTC m=+856.071445936"
Sep 11 16:32:10.578380 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:10.578345    2550 generic.go:356] "Generic (PLEG): container finished" podID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerID="037a0e390fbb49919c273021e87f611f7d8de83b83292992a90dff54bbff1a85" exitCode=0
Sep 11 16:32:10.578380 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:10.578382    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerDied","Data":"037a0e390fbb49919c273021e87f611f7d8de83b83292992a90dff54bbff1a85"}
Sep 11 16:32:11.705657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.705609    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kt8qc"
Sep 11 16:32:11.781501 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.781469    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zffhg\" (UniqueName: \"kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg\") pod \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") "
Sep 11 16:32:11.781682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.781535    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache\") pod \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\" (UID: \"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f\") "
Sep 11 16:32:11.783607 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.783580    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg" (OuterVolumeSpecName: "kube-api-access-zffhg") pod "33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" (UID: "33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f"). InnerVolumeSpecName "kube-api-access-zffhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:32:11.800976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.800952    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" (UID: "33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:32:11.882568 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.882503    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:32:11.882568 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:11.882526    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zffhg\" (UniqueName: \"kubernetes.io/projected/33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f-kube-api-access-zffhg\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:32:12.588820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:12.588786    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kt8qc" event={"ID":"33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f","Type":"ContainerDied","Data":"4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89"}
Sep 11 16:32:12.588820 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:12.588825    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4622e7141390a5d2a63b92b13207f1884dea7a44bc207d04271e434829a42e89"
Sep 11 16:32:12.589029 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:12.588821    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kt8qc"
Sep 11 16:32:12.980191 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:12.980163    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:13.064903 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:13.064866    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:25.073087 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:25.073055    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:25.172861 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:25.172831    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:32.790767 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:32.790727    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:33.164355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:33.164263    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:38.089566 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.089537    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:32:38.090594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.090567    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:38.091426 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.090962    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerName="s3-init"
Sep 11 16:32:38.091426 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.090988    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerName="s3-init"
Sep 11 16:32:38.091426 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.091039    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerName="download-hf-model"
Sep 11 16:32:38.091426 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.091048    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerName="download-hf-model"
Sep 11 16:32:38.091426 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.091231    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f" containerName="s3-init"
Sep 11 16:32:38.095532 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.095512    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:32:38.095653 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.095599    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.098443 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098401    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 11 16:32:38.098581 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098529    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 11 16:32:38.098700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098669    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:32:38.098772 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098719    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 11 16:32:38.098853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098762    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 11 16:32:38.098853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.098793    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-g9xsn\""
Sep 11 16:32:38.203990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.203956    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.203990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.203993    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204240 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204019    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204240 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204037    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204240 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204116    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5mj\" (UniqueName: \"kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204240 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204138    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204240 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204154    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.204432 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.204239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305476 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305440    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5mj\" (UniqueName: \"kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305599    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305647    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.305681 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305680    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306022 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306022 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.305941    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306132 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.306017    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306132 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.306070    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306235 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.306133    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.306298 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.306267    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.307940 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.307919    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.308091 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.308073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.313613 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.313591    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5mj\" (UniqueName: \"kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.406706 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.406634    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:32:38.702722 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:38.702699    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:32:38.705020 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:32:38.704993    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e7b4c_152b_4844_9275_a7313f138690.slice/crio-d766365c738c4209b246effaaa32166f91aa944cab9105dc3fc0cd6809b2f396 WatchSource:0}: Error finding container d766365c738c4209b246effaaa32166f91aa944cab9105dc3fc0cd6809b2f396: Status 404 returned error can't find the container with id d766365c738c4209b246effaaa32166f91aa944cab9105dc3fc0cd6809b2f396
Sep 11 16:32:39.716276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:39.716235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerStarted","Data":"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"}
Sep 11 16:32:39.716768 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:39.716303    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerStarted","Data":"d766365c738c4209b246effaaa32166f91aa944cab9105dc3fc0cd6809b2f396"}
Sep 11 16:32:41.726180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:41.726143    2550 generic.go:356] "Generic (PLEG): container finished" podID="ab4e7b4c-152b-4844-9275-a7313f138690" containerID="e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b" exitCode=0
Sep 11 16:32:41.726180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:41.726173    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerDied","Data":"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"}
Sep 11 16:32:41.727482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:41.727468    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:32:44.758434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.758400    2550 generic.go:356] "Generic (PLEG): container finished" podID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerID="c4f37d9f1dd462906f0f10917d79a03661623e7e9469ed684b0eca72d0c5a6c0" exitCode=0
Sep 11 16:32:44.759152 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.758477    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" event={"ID":"83a68164-af95-4ca7-8e2c-1a55a7e2ce40","Type":"ContainerDied","Data":"c4f37d9f1dd462906f0f10917d79a03661623e7e9469ed684b0eca72d0c5a6c0"}
Sep 11 16:32:44.759152 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.759003    2550 scope.go:117] "RemoveContainer" containerID="c4f37d9f1dd462906f0f10917d79a03661623e7e9469ed684b0eca72d0c5a6c0"
Sep 11 16:32:44.760193 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.760158    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:32:44.760293 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.760273    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:32:44.760344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:44.760329    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:32:45.217671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:45.217637    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:45.778419 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:45.778378    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" event={"ID":"83a68164-af95-4ca7-8e2c-1a55a7e2ce40","Type":"ContainerStarted","Data":"1a1b1d116e88ae94922b409f52878737fac3f60d3b204041d2b7ee6444fb51d9"}
Sep 11 16:32:46.457648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:46.456004    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:32:46.457648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:46.456169    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:32:46.457648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:46.456216    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:32:47.456101 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:47.456038    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:32:47.456608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:47.456116    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:32:48.465647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:48.463090    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:32:48.465647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:48.463158    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:32:49.470650 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:49.465013    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:32:49.470650 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:49.465097    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:32:50.455496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:50.455012    2550 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-68dcb6f84c-svpft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body=
Sep 11 16:32:50.455496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:50.455094    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft" podUID="83a68164-af95-4ca7-8e2c-1a55a7e2ce40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused"
Sep 11 16:32:51.451873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:51.451842    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:32:51.452374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:51.452353    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:32:51.453124 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:32:51.453101    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-68dcb6f84c-svpft"
Sep 11 16:33:05.665878 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:05.665832    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:12.930533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:12.930493    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerStarted","Data":"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"}
Sep 11 16:33:12.955634 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:12.955574    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podStartSLOduration=2.175050214 podStartE2EDuration="34.955556583s" totalImagesPullingTime="30.332678334s" totalInitContainerRuntime="2.447828035s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:12.952890454 +0000 UTC m=+983.727591027" watchObservedRunningTime="2026-09-11 16:33:12.955556583 +0000 UTC m=+983.730257162"
Sep 11 16:33:13.407580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:13.407544    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:33:14.732558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:14.732521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:14.830882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:14.830851    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:18.407358 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:18.407319    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:33:18.408920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:18.408890    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:33:26.501640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:26.501586    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:26.557716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:26.557683    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:28.408035 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:28.407985    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:33:38.407521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:38.407466    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:33:48.411501 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:48.411051    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:33:58.407236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:33:58.407192    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:04.574558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:04.574521    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:05.948597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:05.948568    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:08.407943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:08.407903    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:10.015113 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:10.015076    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:10.218296 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:10.218264    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:11.406278 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:11.406245    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:16.845166 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:16.845131    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:16.964082 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:16.964052    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:17.564970 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:17.564935    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:18.407749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:18.407703    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:28.407188 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:28.407145    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:38.407089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:38.407047    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:48.407590 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:48.407483    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:34:49.568105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:49.568067    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:49.666451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:49.666418    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:34:58.407501 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:34:58.407454    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": dial tcp 10.132.0.28:8000: connect: connection refused"
Sep 11 16:35:08.417590 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:08.417554    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:35:08.425731 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:08.425711    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:35:12.677692 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:12.677654    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:35:12.692745 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:12.692705    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:26.008661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:26.008614    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:27.699675 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:27.699640    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" containerID="cri-o://57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1" gracePeriod=45
Sep 11 16:35:28.442709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.442687    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:35:28.568996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.568919    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.568996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.568962    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.568996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.568986    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569007    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td5mj\" (UniqueName: \"kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569134    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569225    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:28.569275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569229    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569282    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569331    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm\") pod \"ab4e7b4c-152b-4844-9275-a7313f138690\" (UID: \"ab4e7b4c-152b-4844-9275-a7313f138690\") "
Sep 11 16:35:28.569493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569336    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:35:28.569727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569708    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e7b4c-152b-4844-9275-a7313f138690-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.569811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569734    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.570014 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.569988    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home" (OuterVolumeSpecName: "home") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:28.571469 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.571445    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:35:28.571573 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.571551    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj" (OuterVolumeSpecName: "kube-api-access-td5mj") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "kube-api-access-td5mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:35:28.571839 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.571816    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm" (OuterVolumeSpecName: "dshm") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:28.581693 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.581669    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:28.589043 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.589021    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab4e7b4c-152b-4844-9275-a7313f138690" (UID: "ab4e7b4c-152b-4844-9275-a7313f138690"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:28.594709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.594685    2550 generic.go:356] "Generic (PLEG): container finished" podID="ab4e7b4c-152b-4844-9275-a7313f138690" containerID="57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1" exitCode=0
Sep 11 16:35:28.594781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.594733    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"
Sep 11 16:35:28.594835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.594734    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerDied","Data":"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"}
Sep 11 16:35:28.594876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.594853    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" event={"ID":"ab4e7b4c-152b-4844-9275-a7313f138690","Type":"ContainerDied","Data":"d766365c738c4209b246effaaa32166f91aa944cab9105dc3fc0cd6809b2f396"}
Sep 11 16:35:28.594912 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.594883    2550 scope.go:117] "RemoveContainer" containerID="57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"
Sep 11 16:35:28.604040 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.604023    2550 scope.go:117] "RemoveContainer" containerID="e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"
Sep 11 16:35:28.611303 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.611287    2550 scope.go:117] "RemoveContainer" containerID="57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"
Sep 11 16:35:28.611551 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:35:28.611531    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1\": container with ID starting with 57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1 not found: ID does not exist" containerID="57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"
Sep 11 16:35:28.611616 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.611563    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1"} err="failed to get container status \"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1\": rpc error: code = NotFound desc = could not find container \"57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1\": container with ID starting with 57d79666d12d5762f0e1a8d0cef522f98c5adef9d18e0affbd3221571f029ad1 not found: ID does not exist"
Sep 11 16:35:28.611616 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.611587    2550 scope.go:117] "RemoveContainer" containerID="e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"
Sep 11 16:35:28.611886 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:35:28.611866    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b\": container with ID starting with e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b not found: ID does not exist" containerID="e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"
Sep 11 16:35:28.611935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.611891    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b"} err="failed to get container status \"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b\": rpc error: code = NotFound desc = could not find container \"e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b\": container with ID starting with e4de314b38eb365ff4ed40d36672d5d62538ee9ae920155a50ffdb5831582f9b not found: ID does not exist"
Sep 11 16:35:28.617647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.617599    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:35:28.622985 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.622966    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh"]
Sep 11 16:35:28.670770 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670745    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.670770 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670767    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.670913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670778    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.670913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670787    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4e7b4c-152b-4844-9275-a7313f138690-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.670913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670797    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5mj\" (UniqueName: \"kubernetes.io/projected/ab4e7b4c-152b-4844-9275-a7313f138690-kube-api-access-td5mj\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:28.670913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:28.670805    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4e7b4c-152b-4844-9275-a7313f138690-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:35:29.419451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:29.419409    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75987bc644j2bvh" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": context deadline exceeded"
Sep 11 16:35:29.688636 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:29.688549    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" path="/var/lib/kubelet/pods/ab4e7b4c-152b-4844-9275-a7313f138690/volumes"
Sep 11 16:35:32.680097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:32.680059    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:32.777865 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:32.777837    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:39.008253 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.008215    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:39.645215 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.645183    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:39.745277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.745250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:39.859494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859462    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:35:39.859818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859799    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="storage-initializer"
Sep 11 16:35:39.859818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859813    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="storage-initializer"
Sep 11 16:35:39.859959 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859831    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main"
Sep 11 16:35:39.859959 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859839    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main"
Sep 11 16:35:39.859959 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.859941    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4e7b4c-152b-4844-9275-a7313f138690" containerName="main"
Sep 11 16:35:39.863346 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.863328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.866154 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.866051    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 11 16:35:39.866238 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.866193    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:35:39.866297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.866256    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 11 16:35:39.866482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.866468    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-fnlxh\""
Sep 11 16:35:39.877996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.877976    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:35:39.902068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.902006    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:39.966408 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966427    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966451    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966562    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966588    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966653    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966691    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvtt\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:39.966771 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:39.966715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.003206 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.003168    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:40.067896 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.067864    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.067910    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.067932    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068012    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068087    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068126    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvtt\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068360 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068501    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068533    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068572    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068597    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.068836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.068808    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.070223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.070200    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.070553 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.070535    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.076313 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.076293    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvtt\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.076403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.076332    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-8w899\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.175383 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.175297    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:40.449673 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.449651    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:35:40.451249 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:35:40.451229    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daeddc0_d569_443f_b046_c86989710332.slice/crio-dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad WatchSource:0}: Error finding container dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad: Status 404 returned error can't find the container with id dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad
Sep 11 16:35:40.453740 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.453710    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:35:40.453803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.453771    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:35:40.453803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.453796    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:35:40.651122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.651078    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" event={"ID":"6daeddc0-d569-443f-b046-c86989710332","Type":"ContainerStarted","Data":"7ef0e942ccdb458a303b67d6df153a33ad61fc95b09a857e3de8f6ac5b6519cd"}
Sep 11 16:35:40.651312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.651135    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" event={"ID":"6daeddc0-d569-443f-b046-c86989710332","Type":"ContainerStarted","Data":"dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad"}
Sep 11 16:35:40.672167 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:40.672111    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" podStartSLOduration=1.672094628 podStartE2EDuration="1.672094628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:35:40.670304008 +0000 UTC m=+1131.445004587" watchObservedRunningTime="2026-09-11 16:35:40.672094628 +0000 UTC m=+1131.446795206"
Sep 11 16:35:41.175829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:41.175787    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:41.180613 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:41.180587    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:41.656583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:41.656551    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:41.657326 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:41.657307    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:35:43.073882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:43.073835    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:44.585234 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:44.585202    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:44.684868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:44.684836    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:45.948486 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:45.948460    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:51.637321 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:51.637289    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:35:51.702751 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:35:51.702722    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:03.209477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.209446    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:03.320510 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.320474    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:03.547135 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.547038    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:36:03.547406 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.547314    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:03.552080 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.552060    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.556438 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.556417    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:36:03.556661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.556640    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 11 16:36:03.556786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.556706    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 11 16:36:03.557110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.557097    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-v968p\""
Sep 11 16:36:03.557751 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.557728    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 11 16:36:03.557853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.557788    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-st64d\""
Sep 11 16:36:03.558111 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.558087    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 11 16:36:03.571675 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571651    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571778 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571690    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571778 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571778 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571769    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l24g\" (UniqueName: \"kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571845    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571904    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.571950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.571940    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.587330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.587308    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:36:03.667682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.667609    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:03.672756 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672727    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.672888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672762    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.672953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672885    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.672953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672936    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l24g\" (UniqueName: \"kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.672996    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673022    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673069    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673175    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673270 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673222    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673422 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673344    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.673571 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.673555    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.675384 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.675362    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.676223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.676202    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.690161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.690134    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l24g\" (UniqueName: \"kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:03.864594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:03.864508    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:04.143244 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:04.143143    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:36:04.147360 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:36:04.147332    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3010913_3cea_4765_8648_941f2616467e.slice/crio-25ac77152c2dcbc6078846977324993e3f3d86c03e4ab4299d1a278eddceaa87 WatchSource:0}: Error finding container 25ac77152c2dcbc6078846977324993e3f3d86c03e4ab4299d1a278eddceaa87: Status 404 returned error can't find the container with id 25ac77152c2dcbc6078846977324993e3f3d86c03e4ab4299d1a278eddceaa87
Sep 11 16:36:04.759727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:04.759683    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerStarted","Data":"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"}
Sep 11 16:36:04.760218 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:04.759756    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerStarted","Data":"25ac77152c2dcbc6078846977324993e3f3d86c03e4ab4299d1a278eddceaa87"}
Sep 11 16:36:07.783399 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:07.783365    2550 generic.go:356] "Generic (PLEG): container finished" podID="e3010913-3cea-4765-8648-941f2616467e" containerID="0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4" exitCode=0
Sep 11 16:36:07.783873 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:07.783435    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerDied","Data":"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"}
Sep 11 16:36:08.790833 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:08.790802    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerStarted","Data":"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"}
Sep 11 16:36:08.813934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:08.813873    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.243610113 podStartE2EDuration="5.813852136s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.570242023s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:08.812068082 +0000 UTC m=+1159.586768681" watchObservedRunningTime="2026-09-11 16:36:08.813852136 +0000 UTC m=+1159.588552715"
Sep 11 16:36:08.865089 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:08.865051    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:13.864863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:13.864777    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:36:13.866138 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:13.866106    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:36:22.239786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:22.239754    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:23.865042 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:23.864995    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:36:33.865117 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:33.865065    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:36:43.865145 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:43.865092    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:36:47.429809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:47.429767    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:49.713440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.713399    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:49.898816 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.898774    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:49.986535 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.986452    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:36:49.989587 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.989559    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:49.989773 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.989748    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:49.992369 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.992341    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 11 16:36:49.992498 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.992434    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-xwhcm\""
Sep 11 16:36:49.992710 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.992693    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 11 16:36:49.992956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:49.992701    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:36:50.005165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.005139    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:36:50.077196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.077163    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:50.094809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.094776    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.094990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.094853    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.094990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.094915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.094990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.094944    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.094990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.094988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.095152 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.095003    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.095152 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.095022    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.095224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.095158    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vkp\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.095224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.095201    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196762    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196847    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196896    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.196992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.196955    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197044    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vkp\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197076    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197103    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197442 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197283    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197442 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197334    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197442 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197557 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197522    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.197942 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.197919    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.199348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.199327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.199438 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.199360    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.207432 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.207411    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.207552 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.207534    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vkp\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp\") pod \"router-gateway-1-openshift-default-66bbccc5fb-mqtgc\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.304760 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.304673    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:50.606950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.606927    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:36:50.609036 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:36:50.609010    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130f5893_b212_4a21_89c6_e50fb5f74703.slice/crio-eedf88c04a415f2cc4ff7f27c6c7c3fc23b93bf280786d5097c9234be4d185ea WatchSource:0}: Error finding container eedf88c04a415f2cc4ff7f27c6c7c3fc23b93bf280786d5097c9234be4d185ea: Status 404 returned error can't find the container with id eedf88c04a415f2cc4ff7f27c6c7c3fc23b93bf280786d5097c9234be4d185ea
Sep 11 16:36:50.611202 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.611164    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:36:50.611297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.611250    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:36:50.611338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:50.611300    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 16:36:51.007913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:51.007868    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" event={"ID":"130f5893-b212-4a21-89c6-e50fb5f74703","Type":"ContainerStarted","Data":"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"}
Sep 11 16:36:51.008395 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:51.007927    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" event={"ID":"130f5893-b212-4a21-89c6-e50fb5f74703","Type":"ContainerStarted","Data":"eedf88c04a415f2cc4ff7f27c6c7c3fc23b93bf280786d5097c9234be4d185ea"}
Sep 11 16:36:51.061923 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:51.061854    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" podStartSLOduration=2.061834642 podStartE2EDuration="2.061834642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:51.048599342 +0000 UTC m=+1201.823299920" watchObservedRunningTime="2026-09-11 16:36:51.061834642 +0000 UTC m=+1201.836535221"
Sep 11 16:36:51.305595 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:51.305496    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:51.309929 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:51.309906    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:52.012899 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:52.012861    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:52.013800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:52.013781    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:36:52.718057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:52.718024    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:53.865761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:53.865712    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:36:54.690384 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:54.690347    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:36:54.690639 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:54.690586    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" podUID="6daeddc0-d569-443f-b046-c86989710332" containerName="istio-proxy" containerID="cri-o://7ef0e942ccdb458a303b67d6df153a33ad61fc95b09a857e3de8f6ac5b6519cd" gracePeriod=30
Sep 11 16:36:54.778474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:54.778435    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:36:56.795194 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:56.795153    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" podUID="6daeddc0-d569-443f-b046-c86989710332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": read tcp 10.132.0.2:60022->10.132.0.29:15021: read: connection reset by peer"
Sep 11 16:36:57.041638 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.041594    2550 generic.go:356] "Generic (PLEG): container finished" podID="6daeddc0-d569-443f-b046-c86989710332" containerID="7ef0e942ccdb458a303b67d6df153a33ad61fc95b09a857e3de8f6ac5b6519cd" exitCode=0
Sep 11 16:36:57.041769 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.041654    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" event={"ID":"6daeddc0-d569-443f-b046-c86989710332","Type":"ContainerDied","Data":"7ef0e942ccdb458a303b67d6df153a33ad61fc95b09a857e3de8f6ac5b6519cd"}
Sep 11 16:36:57.041769 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.041745    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899" event={"ID":"6daeddc0-d569-443f-b046-c86989710332","Type":"ContainerDied","Data":"dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad"}
Sep 11 16:36:57.041871 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.041795    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd26a6d5c2426b14567c7b7b62218ae93c036bb98f5a7b40dc5e43612bcaf4ad"
Sep 11 16:36:57.044346 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.044298    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:36:57.162048 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162019    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lvtt\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162061    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162102    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162147    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162165    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162190    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162218    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162271    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162322    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data\") pod \"6daeddc0-d569-443f-b046-c86989710332\" (UID: \"6daeddc0-d569-443f-b046-c86989710332\") "
Sep 11 16:36:57.162658 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162480    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:57.162733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162667    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.162733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162665    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:57.162733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162562    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:57.162733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162702    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:57.162936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.162853    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data" (OuterVolumeSpecName: "istio-data") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:57.164409 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.164376    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 16:36:57.164528 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.164508    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:57.164595 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.164561    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token" (OuterVolumeSpecName: "istio-token") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:57.164664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.164591    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt" (OuterVolumeSpecName: "kube-api-access-5lvtt") pod "6daeddc0-d569-443f-b046-c86989710332" (UID: "6daeddc0-d569-443f-b046-c86989710332"). InnerVolumeSpecName "kube-api-access-5lvtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:57.263888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263854    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-data\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.263888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263883    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvtt\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-kube-api-access-5lvtt\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263896    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6daeddc0-d569-443f-b046-c86989710332-istio-podinfo\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263908    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-istio-envoy\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263919    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6daeddc0-d569-443f-b046-c86989710332-istiod-ca-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263930    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6daeddc0-d569-443f-b046-c86989710332-istio-token\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263942    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-workload-socket\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:57.264088 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:57.263953    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6daeddc0-d569-443f-b046-c86989710332-credential-socket\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:36:58.046633 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:58.046601    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"
Sep 11 16:36:58.065543 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:58.065505    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:36:58.069200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:58.069176    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-8w899"]
Sep 11 16:36:59.688671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:36:59.688638    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6daeddc0-d569-443f-b046-c86989710332" path="/var/lib/kubelet/pods/6daeddc0-d569-443f-b046-c86989710332/volumes"
Sep 11 16:37:01.065130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:01.065094    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:01.280786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:01.280749    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:01.405020 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:01.404945    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:03.864993 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:03.864942    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:37:05.244147 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:05.244107    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:06.307516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:06.307479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:06.408267 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:06.408238    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:06.831116 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:06.831075    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:12.926846 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:12.926809    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:13.865175 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:13.865140    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:37:15.232797 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:15.232758    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:15.321761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:15.321727    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:16.626186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:16.626150    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:16.727231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:16.727192    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:17.439977 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:17.439942    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:20.195923 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:20.195889    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:37:20.196261 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:20.196145    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" containerName="istio-proxy" containerID="cri-o://97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e" gracePeriod=30
Sep 11 16:37:20.246075 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:20.246039    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:22.184131 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:22.184074    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:22.248086 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:22.248047    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:23.053784 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.053742    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 16:37:23.348285 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.348252    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:37:23.513152 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513121    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513159    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513200    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513216    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513258    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513287    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513304    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4vkp\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513354    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513388    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy\") pod \"130f5893-b212-4a21-89c6-e50fb5f74703\" (UID: \"130f5893-b212-4a21-89c6-e50fb5f74703\") "
Sep 11 16:37:23.513668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513409    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:23.513668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513417    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data" (OuterVolumeSpecName: "istio-data") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:23.513668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513650    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:23.513936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513668    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:23.513936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513695    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-data\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.513936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513713    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-credential-socket\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.513936 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.513783    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:37:23.515588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.515564    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token" (OuterVolumeSpecName: "istio-token") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:37:23.515986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.515967    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 16:37:23.516078 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.515986    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp" (OuterVolumeSpecName: "kube-api-access-p4vkp") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "kube-api-access-p4vkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:37:23.516204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.516188    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "130f5893-b212-4a21-89c6-e50fb5f74703" (UID: "130f5893-b212-4a21-89c6-e50fb5f74703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614880    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-socket\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614905    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-istio-token\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614915    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vkp\" (UniqueName: \"kubernetes.io/projected/130f5893-b212-4a21-89c6-e50fb5f74703-kube-api-access-p4vkp\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614924    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/130f5893-b212-4a21-89c6-e50fb5f74703-istio-podinfo\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614933    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-istio-envoy\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614942    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/130f5893-b212-4a21-89c6-e50fb5f74703-istiod-ca-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.614953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.614949    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/130f5893-b212-4a21-89c6-e50fb5f74703-workload-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:37:23.865602 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:23.865498    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:37:24.182464 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.182352    2550 generic.go:356] "Generic (PLEG): container finished" podID="130f5893-b212-4a21-89c6-e50fb5f74703" containerID="97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e" exitCode=0
Sep 11 16:37:24.182464 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.182397    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"
Sep 11 16:37:24.182464 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.182427    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" event={"ID":"130f5893-b212-4a21-89c6-e50fb5f74703","Type":"ContainerDied","Data":"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"}
Sep 11 16:37:24.182847 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.182485    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc" event={"ID":"130f5893-b212-4a21-89c6-e50fb5f74703","Type":"ContainerDied","Data":"eedf88c04a415f2cc4ff7f27c6c7c3fc23b93bf280786d5097c9234be4d185ea"}
Sep 11 16:37:24.182847 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.182509    2550 scope.go:117] "RemoveContainer" containerID="97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"
Sep 11 16:37:24.193491 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.193461    2550 scope.go:117] "RemoveContainer" containerID="97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"
Sep 11 16:37:24.193829 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:37:24.193807    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e\": container with ID starting with 97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e not found: ID does not exist" containerID="97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"
Sep 11 16:37:24.193934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.193835    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e"} err="failed to get container status \"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e\": rpc error: code = NotFound desc = could not find container \"97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e\": container with ID starting with 97cad5d3425312e5f6d750590cffe5b80271073c9327162bbef4bc564d82724e not found: ID does not exist"
Sep 11 16:37:24.204686 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.204612    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:37:24.209533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:24.209495    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-mqtgc"]
Sep 11 16:37:25.319543 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:25.319505    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:25.691192 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:25.690726    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" path="/var/lib/kubelet/pods/130f5893-b212-4a21-89c6-e50fb5f74703/volumes"
Sep 11 16:37:26.278706 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:26.278579    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:28.483417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:28.483374    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:28.544391 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:28.544360    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:33.865938 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:33.865891    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:37:37.680129 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:37.680097    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:38.695190 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695152    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:37:38.695735 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695703    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" containerName="istio-proxy"
Sep 11 16:37:38.695735 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695721    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" containerName="istio-proxy"
Sep 11 16:37:38.695876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695743    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6daeddc0-d569-443f-b046-c86989710332" containerName="istio-proxy"
Sep 11 16:37:38.695876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695754    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6daeddc0-d569-443f-b046-c86989710332" containerName="istio-proxy"
Sep 11 16:37:38.695876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695848    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="130f5893-b212-4a21-89c6-e50fb5f74703" containerName="istio-proxy"
Sep 11 16:37:38.695876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.695860    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6daeddc0-d569-443f-b046-c86989710332" containerName="istio-proxy"
Sep 11 16:37:38.702821 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.702797    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.706101 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706070    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 11 16:37:38.706249 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706166    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 11 16:37:38.706249 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706199    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:37:38.706355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706257    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 11 16:37:38.706355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706166    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-hhjln\""
Sep 11 16:37:38.707005 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.706988    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 11 16:37:38.712788 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.712766    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:37:38.734348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.734325    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:38.761090 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761057    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjjb\" (UniqueName: \"kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761217 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761106    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761217 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761146    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761217 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761179    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761219    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761279    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761410 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.761596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.761434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862372 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862336    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862372 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862385    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862415    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862485    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862562    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjjb\" (UniqueName: \"kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862596    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862649    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.862937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.863056 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.862957    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.863056 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.863005    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.863265 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.863243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.863564 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.863539    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.863744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.863656    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.866139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.866096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.866395 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.866371    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.871824 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.871805    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjjb\" (UniqueName: \"kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb\") pod \"auth-enabled-test-kserve-77f6f8dc8f-xqfff\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:38.974493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:38.974411    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:39.015394 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:39.015370    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:39.299665 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:39.299617    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:37:39.301088 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:37:39.301054    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d79d896_b037_46b5_823d_31c3766dafde.slice/crio-027846bf285a1890022bc57330e849b7a48f97699940107fa1e4313c8fc75e7d WatchSource:0}: Error finding container 027846bf285a1890022bc57330e849b7a48f97699940107fa1e4313c8fc75e7d: Status 404 returned error can't find the container with id 027846bf285a1890022bc57330e849b7a48f97699940107fa1e4313c8fc75e7d
Sep 11 16:37:40.289806 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:40.289766    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerStarted","Data":"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"}
Sep 11 16:37:40.290512 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:40.289825    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerStarted","Data":"027846bf285a1890022bc57330e849b7a48f97699940107fa1e4313c8fc75e7d"}
Sep 11 16:37:40.877277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:40.877239    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:40.978906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:40.978869    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:37:42.302506 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:42.302433    2550 generic.go:356] "Generic (PLEG): container finished" podID="3d79d896-b037-46b5-823d-31c3766dafde" containerID="4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba" exitCode=0
Sep 11 16:37:42.302959 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:42.302503    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerDied","Data":"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"}
Sep 11 16:37:43.309890 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:43.309856    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerStarted","Data":"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"}
Sep 11 16:37:43.340187 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:43.340143    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podStartSLOduration=3.149118771 podStartE2EDuration="5.340127194s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.191008423s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:37:43.338297095 +0000 UTC m=+1254.112997675" watchObservedRunningTime="2026-09-11 16:37:43.340127194 +0000 UTC m=+1254.114827771"
Sep 11 16:37:43.875127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:43.875097    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:37:43.884433 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:43.884402    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:37:44.016217 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:44.016189    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:49.016330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:49.016298    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:37:49.017579 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:49.017551    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:37:59.016719 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:37:59.016672    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:05.673599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:05.673566    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:09.016226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:09.016175    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:12.442313 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:12.442281    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:12.460657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:12.459930    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:38:12.506882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:12.506850    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:19.016291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:19.016248    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:27.500385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:27.500333    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" containerID="cri-o://d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc" gracePeriod=45
Sep 11 16:38:27.876687 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:27.876573    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 11 16:38:28.143101 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.143076    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:38:28.210461 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210425    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210472    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210503    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l24g\" (UniqueName: \"kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210565    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210652 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210605    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210664    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210701    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210739    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert\") pod \"e3010913-3cea-4765-8648-941f2616467e\" (UID: \"e3010913-3cea-4765-8648-941f2616467e\") "
Sep 11 16:38:28.210892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.210785    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:28.211098 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.211050    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.211408 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.211383    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:28.211523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.211409    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home" (OuterVolumeSpecName: "home") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:28.213427 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.213387    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g" (OuterVolumeSpecName: "kube-api-access-8l24g") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "kube-api-access-8l24g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:28.213588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.213502    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm" (OuterVolumeSpecName: "dshm") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:28.213588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.213560    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:28.224036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.224004    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:28.233973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.233937    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3010913-3cea-4765-8648-941f2616467e" (UID: "e3010913-3cea-4765-8648-941f2616467e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:28.311494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311453    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3010913-3cea-4765-8648-941f2616467e-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311480    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311489    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l24g\" (UniqueName: \"kubernetes.io/projected/e3010913-3cea-4765-8648-941f2616467e-kube-api-access-8l24g\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311499    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311508    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311516    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3010913-3cea-4765-8648-941f2616467e-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.311952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.311525    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3010913-3cea-4765-8648-941f2616467e-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:38:28.563382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.563346    2550 generic.go:356] "Generic (PLEG): container finished" podID="e3010913-3cea-4765-8648-941f2616467e" containerID="d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc" exitCode=0
Sep 11 16:38:28.563901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.563394    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 11 16:38:28.563901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.563425    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerDied","Data":"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"}
Sep 11 16:38:28.563901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.563477    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e3010913-3cea-4765-8648-941f2616467e","Type":"ContainerDied","Data":"25ac77152c2dcbc6078846977324993e3f3d86c03e4ab4299d1a278eddceaa87"}
Sep 11 16:38:28.563901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.563500    2550 scope.go:117] "RemoveContainer" containerID="d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"
Sep 11 16:38:28.572947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.572930    2550 scope.go:117] "RemoveContainer" containerID="0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"
Sep 11 16:38:28.581747 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.581731    2550 scope.go:117] "RemoveContainer" containerID="d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"
Sep 11 16:38:28.582003 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:38:28.581986    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc\": container with ID starting with d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc not found: ID does not exist" containerID="d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"
Sep 11 16:38:28.582051 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.582012    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc"} err="failed to get container status \"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc\": rpc error: code = NotFound desc = could not find container \"d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc\": container with ID starting with d064625cb3adeee18befcecef9854ad106412fb3cf36c5740208e0fe6dacbbfc not found: ID does not exist"
Sep 11 16:38:28.582051 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.582029    2550 scope.go:117] "RemoveContainer" containerID="0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"
Sep 11 16:38:28.582277 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:38:28.582260    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4\": container with ID starting with 0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4 not found: ID does not exist" containerID="0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"
Sep 11 16:38:28.582345 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.582285    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4"} err="failed to get container status \"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4\": rpc error: code = NotFound desc = could not find container \"0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4\": container with ID starting with 0b804f1ef713688e4f1161bb05df9e73cdfc87084aa5c8e6ca9cc18d0d2b77f4 not found: ID does not exist"
Sep 11 16:38:28.587707 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.587679    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:38:28.591999 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:28.591979    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 11 16:38:29.016666 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:29.016600    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:29.688901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:29.688867    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3010913-3cea-4765-8648-941f2616467e" path="/var/lib/kubelet/pods/e3010913-3cea-4765-8648-941f2616467e/volumes"
Sep 11 16:38:39.015964 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:39.015919    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:43.361514 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:43.361478    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:43.462998 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:43.462966    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:38:49.015926 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:49.015880    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:38:59.015966 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:38:59.015916    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:39:02.611380 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:02.611343    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:02.703299 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:02.703263    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:08.713579 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:08.713541    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:09.016069 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:09.015987    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:39:14.454932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:14.454825    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:14.528296 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:14.528250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:19.016361 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:19.016319    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:39:23.988191 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:23.988147    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:24.172232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:24.172196    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:29.016182 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:29.016142    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:39:33.572789 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:33.572754    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:39.026248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:39.026219    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:39:39.033990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:39.033967    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:39:43.508534 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:43.508498    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:43.509284 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:43.509254    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:39:43.568580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:43.568539    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:57.710080 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:57.710042    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:57.809291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:57.809256    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:39:58.531595 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:58.531552    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" containerID="cri-o://f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea" gracePeriod=45
Sep 11 16:39:59.026800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.026765    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Sep 11 16:39:59.280922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.280864    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:39:59.413648 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413608    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.413790 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413652    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.413790 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413693    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.413790 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413722    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.413790 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413784    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.414001 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413843    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.414001 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413870    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsjjb\" (UniqueName: \"kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.414001 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.413894    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm\") pod \"3d79d896-b037-46b5-823d-31c3766dafde\" (UID: \"3d79d896-b037-46b5-823d-31c3766dafde\") "
Sep 11 16:39:59.414161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.414113    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:59.414268 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.414235    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:39:59.414463 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.414437    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home" (OuterVolumeSpecName: "home") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:59.416248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.416220    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb" (OuterVolumeSpecName: "kube-api-access-vsjjb") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "kube-api-access-vsjjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:39:59.416248 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.416237    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:39:59.416433 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.416415    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm" (OuterVolumeSpecName: "dshm") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:59.426894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.426868    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:59.440198 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.440168    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d79d896-b037-46b5-823d-31c3766dafde" (UID: "3d79d896-b037-46b5-823d-31c3766dafde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:59.515508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515483    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515504    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d79d896-b037-46b5-823d-31c3766dafde-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515514    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515523    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515532    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjjb\" (UniqueName: \"kubernetes.io/projected/3d79d896-b037-46b5-823d-31c3766dafde-kube-api-access-vsjjb\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515540    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515547    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79d896-b037-46b5-823d-31c3766dafde-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:39:59.515703 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:39:59.515555    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79d896-b037-46b5-823d-31c3766dafde-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:40:00.019671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.019565    2550 generic.go:356] "Generic (PLEG): container finished" podID="3d79d896-b037-46b5-823d-31c3766dafde" containerID="f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea" exitCode=0
Sep 11 16:40:00.019671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.019612    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"
Sep 11 16:40:00.019857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.019610    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerDied","Data":"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"}
Sep 11 16:40:00.019857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.019761    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff" event={"ID":"3d79d896-b037-46b5-823d-31c3766dafde","Type":"ContainerDied","Data":"027846bf285a1890022bc57330e849b7a48f97699940107fa1e4313c8fc75e7d"}
Sep 11 16:40:00.019857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.019790    2550 scope.go:117] "RemoveContainer" containerID="f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"
Sep 11 16:40:00.028039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.028019    2550 scope.go:117] "RemoveContainer" containerID="4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"
Sep 11 16:40:00.035278 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.035261    2550 scope.go:117] "RemoveContainer" containerID="f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"
Sep 11 16:40:00.035546 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:40:00.035522    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea\": container with ID starting with f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea not found: ID does not exist" containerID="f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"
Sep 11 16:40:00.035604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.035557    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea"} err="failed to get container status \"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea\": rpc error: code = NotFound desc = could not find container \"f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea\": container with ID starting with f2f69edaeeb49719a1c68d49d14ee93b0794a4ecf6b8ed96692193573f56d2ea not found: ID does not exist"
Sep 11 16:40:00.035604 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.035576    2550 scope.go:117] "RemoveContainer" containerID="4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"
Sep 11 16:40:00.035834 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:40:00.035817    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba\": container with ID starting with 4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba not found: ID does not exist" containerID="4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"
Sep 11 16:40:00.035876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.035839    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba"} err="failed to get container status \"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba\": rpc error: code = NotFound desc = could not find container \"4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba\": container with ID starting with 4cd04b621f4539d927d88c1e3378399e33927afc6cadcc142bb1c1d4cf8336ba not found: ID does not exist"
Sep 11 16:40:00.039545 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.039524    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:40:00.043777 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:00.043758    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-77f6f8dc8f-xqfff"]
Sep 11 16:40:01.686257 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:01.686223    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d79d896-b037-46b5-823d-31c3766dafde" path="/var/lib/kubelet/pods/3d79d896-b037-46b5-823d-31c3766dafde/volumes"
Sep 11 16:40:33.419647 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:33.415865    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:33.511937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:33.511900    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:39.434601 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:39.434567    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:43.317436 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:43.317359    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:43.714104 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:43.714059    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:45.363180 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:45.363139    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:45.462770 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:45.462734    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:47.791709 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:47.791676    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:47.887044 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:47.887006    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:53.880066 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:53.880028    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:59.478390 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:59.478352    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:40:59.549314 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:40:59.549279    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:01.178977 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:01.178940    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:07.069455 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069419    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069835    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069855    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069869    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="storage-initializer"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069877    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="storage-initializer"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069902    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="storage-initializer"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069911    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="storage-initializer"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069928    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main"
Sep 11 16:41:07.070003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.069936    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main"
Sep 11 16:41:07.070291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.070022    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3010913-3cea-4765-8648-941f2616467e" containerName="main"
Sep 11 16:41:07.070291 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.070029    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d79d896-b037-46b5-823d-31c3766dafde" containerName="main"
Sep 11 16:41:07.074272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.074255    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.077057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.077033    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 11 16:41:07.087682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.079017    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 11 16:41:07.087682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.079357    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-lztl9\""
Sep 11 16:41:07.087682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.079754    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 11 16:41:07.090235 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.090210    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:07.102305 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.102282    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:07.178338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6pw\" (UniqueName: \"kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.178435 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.178435 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178382    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.178521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178444    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.178521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178474    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.178521 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.178505    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279483 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279443    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279487    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279547    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279606    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6pw\" (UniqueName: \"kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279863 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.279973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.279959    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.280057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.280036    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.281842 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.281816    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.282016 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.281996    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.288021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.287989    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6pw\" (UniqueName: \"kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.296351 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.296328    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:07.396378 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.396314    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:07.669583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.669560    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:07.671973 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:41:07.671949    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59367fb_d896_44e9_95ff_94705d5f31a6.slice/crio-e4944b8aa90231149917d2d06b8efd076066cc47d7fcead5bb0543a2e640b9dd WatchSource:0}: Error finding container e4944b8aa90231149917d2d06b8efd076066cc47d7fcead5bb0543a2e640b9dd: Status 404 returned error can't find the container with id e4944b8aa90231149917d2d06b8efd076066cc47d7fcead5bb0543a2e640b9dd
Sep 11 16:41:07.674249 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:07.674232    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:41:08.333738 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:08.333699    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" event={"ID":"c59367fb-d896-44e9-95ff-94705d5f31a6","Type":"ContainerStarted","Data":"e4944b8aa90231149917d2d06b8efd076066cc47d7fcead5bb0543a2e640b9dd"}
Sep 11 16:41:09.341548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:09.341514    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" event={"ID":"c59367fb-d896-44e9-95ff-94705d5f31a6","Type":"ContainerStarted","Data":"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"}
Sep 11 16:41:09.358632 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:09.358583    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" podStartSLOduration=0.905877141 podStartE2EDuration="2.358566941s" totalImagesPullingTime="1.4526898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:41:09.357015225 +0000 UTC m=+1460.131715804" watchObservedRunningTime="2026-09-11 16:41:09.358566941 +0000 UTC m=+1460.133267519"
Sep 11 16:41:09.397465 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:09.397400    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:17.396514 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:17.396476    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:17.409227 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:17.409199    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:18.403390 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:18.403363    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:31.977125 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:31.977093    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:41.419992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.419957    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:41.427642 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.427584    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:41.437854 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:41:41.437810    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:41:41.437854 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:41:41.437854 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:41:41.437854 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6" cmd=["/bin/sleep","15"]
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.437899    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" message="" err=<
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:41:41.437917    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerName="main" containerID="cri-o://6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"
Sep 11 16:41:41.438114 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.437975    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerName="main" containerID="cri-o://6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6" gracePeriod=60
Sep 11 16:41:41.516640 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.516593    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:41.787726 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.787705    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:41.868687 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868663    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.868849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868718    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.868849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868743    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.868849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.868849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868788    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.868849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868824    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n6pw\" (UniqueName: \"kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw\") pod \"c59367fb-d896-44e9-95ff-94705d5f31a6\" (UID: \"c59367fb-d896-44e9-95ff-94705d5f31a6\") "
Sep 11 16:41:41.869127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868956    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:41.869184 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.869121    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home" (OuterVolumeSpecName: "home") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:41.869231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.869180    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:41.869231 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.868924    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:41.870991 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.870960    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:41:41.871098 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.871009    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm" (OuterVolumeSpecName: "dshm") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:41.871098 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.871039    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw" (OuterVolumeSpecName: "kube-api-access-4n6pw") pod "c59367fb-d896-44e9-95ff-94705d5f31a6" (UID: "c59367fb-d896-44e9-95ff-94705d5f31a6"). InnerVolumeSpecName "kube-api-access-4n6pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:41:41.970411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.970334    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:41.970411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.970360    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:41.970411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.970373    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c59367fb-d896-44e9-95ff-94705d5f31a6-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:41.970411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.970383    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c59367fb-d896-44e9-95ff-94705d5f31a6-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:41.970411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:41.970394    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6pw\" (UniqueName: \"kubernetes.io/projected/c59367fb-d896-44e9-95ff-94705d5f31a6-kube-api-access-4n6pw\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:41:42.503608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.503566    2550 generic.go:356] "Generic (PLEG): container finished" podID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerID="6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6" exitCode=0
Sep 11 16:41:42.504057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.503633    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"
Sep 11 16:41:42.504057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.503648    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" event={"ID":"c59367fb-d896-44e9-95ff-94705d5f31a6","Type":"ContainerDied","Data":"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"}
Sep 11 16:41:42.504057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.503700    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs" event={"ID":"c59367fb-d896-44e9-95ff-94705d5f31a6","Type":"ContainerDied","Data":"e4944b8aa90231149917d2d06b8efd076066cc47d7fcead5bb0543a2e640b9dd"}
Sep 11 16:41:42.504057 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.503730    2550 scope.go:117] "RemoveContainer" containerID="6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"
Sep 11 16:41:42.512757 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.512736    2550 scope.go:117] "RemoveContainer" containerID="6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"
Sep 11 16:41:42.513007 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:41:42.512986    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6\": container with ID starting with 6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6 not found: ID does not exist" containerID="6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"
Sep 11 16:41:42.513060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.513015    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6"} err="failed to get container status \"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6\": rpc error: code = NotFound desc = could not find container \"6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6\": container with ID starting with 6d1d92d345ac1e7002173c3f86aaf631200bd65006c87534516c37fd02f59fe6 not found: ID does not exist"
Sep 11 16:41:42.524762 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.524737    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:42.533229 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:42.530879    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94flmqvs"]
Sep 11 16:41:43.686950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:43.686917    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" path="/var/lib/kubelet/pods/c59367fb-d896-44e9-95ff-94705d5f31a6/volumes"
Sep 11 16:41:45.218733 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:45.218705    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:41:49.753447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:41:49.753413    2550 scope.go:117] "RemoveContainer" containerID="7ef0e942ccdb458a303b67d6df153a33ad61fc95b09a857e3de8f6ac5b6519cd"
Sep 11 16:42:16.999910 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:16.999823    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:30.830975 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:30.830887    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:30.929393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:30.929350    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:36.688765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:36.688726    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:36.873165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:36.873135    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:42.398293 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:42.398250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:42.456053 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:42.456028    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:49.384139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.384108    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:42:49.384906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.384583    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerName="main"
Sep 11 16:42:49.384906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.384598    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerName="main"
Sep 11 16:42:49.384906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.384682    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59367fb-d896-44e9-95ff-94705d5f31a6" containerName="main"
Sep 11 16:42:49.388003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.387980    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.390743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.390719    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 11 16:42:49.390913 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.390885    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 11 16:42:49.392175 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.392153    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 11 16:42:49.392287 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.392196    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-lb5jw\""
Sep 11 16:42:49.397816 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.397795    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:42:49.416724 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.416697    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:49.516947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.516921    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.517068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.516955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.517068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.516977    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.517068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.517001    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.517200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.517104    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8g5\" (UniqueName: \"kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.517200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.517156    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618505    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618563    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618603    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8g5\" (UniqueName: \"kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618653    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618749 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.618990 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.618969    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.619060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.619041    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.619097 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.619069    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.620794 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.620771    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.621405 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.621390    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 11 16:42:49.626449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.626420    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:42:49.627607 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.627585    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 11 16:42:49.630919 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.630897    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.638266 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.638216    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 11 16:42:49.648669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.648645    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8g5\" (UniqueName: \"kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.702818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.702794    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-lb5jw\""
Sep 11 16:42:49.710048 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.710031    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:49.982494 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:49.982466    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:42:49.984095 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:42:49.984070    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05350af_8c05_4b6b_a256_bb3b4a917bdb.slice/crio-7b0523aad1aefe9bfc60405ab0e78e5a1834be8ead074ffe528e62c4d8d9431e WatchSource:0}: Error finding container 7b0523aad1aefe9bfc60405ab0e78e5a1834be8ead074ffe528e62c4d8d9431e: Status 404 returned error can't find the container with id 7b0523aad1aefe9bfc60405ab0e78e5a1834be8ead074ffe528e62c4d8d9431e
Sep 11 16:42:50.812477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:50.812439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" event={"ID":"a05350af-8c05-4b6b-a256-bb3b4a917bdb","Type":"ContainerStarted","Data":"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"}
Sep 11 16:42:50.812964 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:50.812494    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" event={"ID":"a05350af-8c05-4b6b-a256-bb3b4a917bdb","Type":"ContainerStarted","Data":"7b0523aad1aefe9bfc60405ab0e78e5a1834be8ead074ffe528e62c4d8d9431e"}
Sep 11 16:42:50.830401 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:50.830355    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" podStartSLOduration=1.830339102 podStartE2EDuration="1.830339102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:42:50.829447425 +0000 UTC m=+1561.604148005" watchObservedRunningTime="2026-09-11 16:42:50.830339102 +0000 UTC m=+1561.605039680"
Sep 11 16:42:51.710441 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:51.710409    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:59.710280 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:59.710249    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:59.722326 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:59.722303    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:42:59.861382 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:42:59.861351    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:43:02.746109 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:02.746078    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:03.574731 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:03.574695    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:04.725165 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:04.725127    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:04.957679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:04.957650    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:05.055448 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:05.055352    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:15.777450 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:15.777407    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:23.909617 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:23.909581    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:43:23.920305 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:43:23.920270    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:43:23.920305 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:43:23.920305 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:43:23.920305 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d" cmd=["/bin/sleep","15"]
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:23.920323    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" message="" err=<
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:43:23.920337    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerName="main" containerID="cri-o://aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"
Sep 11 16:43:23.920556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:23.920378    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerName="main" containerID="cri-o://aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d" gracePeriod=60
Sep 11 16:43:23.923301 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:23.923204    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:24.021687 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.021649    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:24.275748 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.275722    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:43:24.308374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308347    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308383    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308428    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308480    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308513 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308506    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw8g5\" (UniqueName: \"kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308538    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache\") pod \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\" (UID: \"a05350af-8c05-4b6b-a256-bb3b4a917bdb\") "
Sep 11 16:43:24.308761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308618    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:24.308761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308744    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home" (OuterVolumeSpecName: "home") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:24.308912 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308834    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:24.308912 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308911    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.309028 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.308923    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.310720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.310700    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:24.310803 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.310763    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5" (OuterVolumeSpecName: "kube-api-access-nw8g5") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "kube-api-access-nw8g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:24.310841 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.310797    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm" (OuterVolumeSpecName: "dshm") pod "a05350af-8c05-4b6b-a256-bb3b4a917bdb" (UID: "a05350af-8c05-4b6b-a256-bb3b4a917bdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:24.409825 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.409797    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05350af-8c05-4b6b-a256-bb3b4a917bdb-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.409825 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.409821    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.409825 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.409830    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8g5\" (UniqueName: \"kubernetes.io/projected/a05350af-8c05-4b6b-a256-bb3b4a917bdb-kube-api-access-nw8g5\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.410043 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.409839    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05350af-8c05-4b6b-a256-bb3b4a917bdb-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:43:24.964548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.964513    2550 generic.go:356] "Generic (PLEG): container finished" podID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerID="aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d" exitCode=0
Sep 11 16:43:24.964976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.964556    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"
Sep 11 16:43:24.964976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.964558    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" event={"ID":"a05350af-8c05-4b6b-a256-bb3b4a917bdb","Type":"ContainerDied","Data":"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"}
Sep 11 16:43:24.964976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.964697    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct" event={"ID":"a05350af-8c05-4b6b-a256-bb3b4a917bdb","Type":"ContainerDied","Data":"7b0523aad1aefe9bfc60405ab0e78e5a1834be8ead074ffe528e62c4d8d9431e"}
Sep 11 16:43:24.964976 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.964725    2550 scope.go:117] "RemoveContainer" containerID="aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"
Sep 11 16:43:24.974192 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.974175    2550 scope.go:117] "RemoveContainer" containerID="aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"
Sep 11 16:43:24.974471 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:43:24.974453    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d\": container with ID starting with aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d not found: ID does not exist" containerID="aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"
Sep 11 16:43:24.974525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.974478    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d"} err="failed to get container status \"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d\": rpc error: code = NotFound desc = could not find container \"aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d\": container with ID starting with aba046a18ad1874341ba44124e48cf80fe5c5e27671163a85a6f61549dcd333d not found: ID does not exist"
Sep 11 16:43:24.990464 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.990444    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:43:24.997939 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:24.997916    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-dn2ct"]
Sep 11 16:43:25.686842 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:25.686809    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" path="/var/lib/kubelet/pods/a05350af-8c05-4b6b-a256-bb3b4a917bdb/volumes"
Sep 11 16:43:53.219212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:53.219167    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:53.320322 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:53.320282    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:43:59.219918 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:43:59.219882    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:04.224956 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:04.224916    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:04.286871 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:04.286839    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:13.975964 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:13.975930    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:15.259901 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:15.259853    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:20.105185 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:20.105146    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:20.212550 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:20.212513    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:20.428660 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:20.428556    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:25.759403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:25.759335    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:25.860186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:25.860093    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:34.785867 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.785828    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:44:34.786337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.786209    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerName="main"
Sep 11 16:44:34.786337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.786221    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerName="main"
Sep 11 16:44:34.786337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.786290    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05350af-8c05-4b6b-a256-bb3b4a917bdb" containerName="main"
Sep 11 16:44:34.789236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.789215    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.791910 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.791888    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 11 16:44:34.792023 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.791888    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-p7nmf\""
Sep 11 16:44:34.793025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.793007    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 11 16:44:34.793126 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.793049    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 11 16:44:34.799702 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.799682    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:44:34.818609 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.818583    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:34.878829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878802    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.878829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878828    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.878989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878851    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.878989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878913    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.878989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878927    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.878989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.878951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qstr6\" (UniqueName: \"kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980137 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980103    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980137 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980137    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980332 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980161    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qstr6\" (UniqueName: \"kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980332 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980242    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980332 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980265    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980332 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980294    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980570 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980547    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980618 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980573    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.980723 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.980705    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.982568 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.982542    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.982870 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.982852    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:34.988927 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:34.988905    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qstr6\" (UniqueName: \"kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-55lvs\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:35.028460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.028436    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:44:35.031934 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.031916    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.034485 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.034466    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-f7v8n\""
Sep 11 16:44:35.042338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.042318    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:44:35.055385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.055363    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:35.081612 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.081591    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.081729 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.081714    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5fx\" (UniqueName: \"kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.102013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.101995    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:35.182674 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.182639    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5fx\" (UniqueName: \"kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.182816 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.182734    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.185176 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.185146    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.190225 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.190200    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5fx\" (UniqueName: \"kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.344802 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.344722    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:35.374266 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.374236    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:44:35.377508 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:44:35.377486    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388f38ac_b453_47dd_84c9_c0a827f4c982.slice/crio-2e16de16d82d489d801255ebe4cc78bcc4792528723b769dc955d55eeafafffb WatchSource:0}: Error finding container 2e16de16d82d489d801255ebe4cc78bcc4792528723b769dc955d55eeafafffb: Status 404 returned error can't find the container with id 2e16de16d82d489d801255ebe4cc78bcc4792528723b769dc955d55eeafafffb
Sep 11 16:44:35.623876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:35.623802    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:44:35.627109 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:44:35.627078    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fcb2ec_b769_4edf_aa8f_9da2237afc8d.slice/crio-7550ca4fc252b7d5647b596ef77a082a2487bbcc0fe1f017b1fc9ed8c2568d83 WatchSource:0}: Error finding container 7550ca4fc252b7d5647b596ef77a082a2487bbcc0fe1f017b1fc9ed8c2568d83: Status 404 returned error can't find the container with id 7550ca4fc252b7d5647b596ef77a082a2487bbcc0fe1f017b1fc9ed8c2568d83
Sep 11 16:44:36.300879 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:36.300826    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" event={"ID":"62fcb2ec-b769-4edf-aa8f-9da2237afc8d","Type":"ContainerStarted","Data":"7550ca4fc252b7d5647b596ef77a082a2487bbcc0fe1f017b1fc9ed8c2568d83"}
Sep 11 16:44:36.303417 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:36.303383    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" event={"ID":"388f38ac-b453-47dd-84c9-c0a827f4c982","Type":"ContainerStarted","Data":"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"}
Sep 11 16:44:36.303531 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:36.303439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" event={"ID":"388f38ac-b453-47dd-84c9-c0a827f4c982","Type":"ContainerStarted","Data":"2e16de16d82d489d801255ebe4cc78bcc4792528723b769dc955d55eeafafffb"}
Sep 11 16:44:36.324166 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:36.324123    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" podStartSLOduration=2.324105742 podStartE2EDuration="2.324105742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:36.322206389 +0000 UTC m=+1667.096906965" watchObservedRunningTime="2026-09-11 16:44:36.324105742 +0000 UTC m=+1667.098806320"
Sep 11 16:44:37.102769 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:37.102738    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:38.315727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:38.315685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" event={"ID":"62fcb2ec-b769-4edf-aa8f-9da2237afc8d","Type":"ContainerStarted","Data":"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"}
Sep 11 16:44:38.316162 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:38.315774    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:44:38.331795 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:38.331716    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" podStartSLOduration=0.885098384 podStartE2EDuration="3.331701881s" totalImagesPullingTime="2.446603497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:38.330700699 +0000 UTC m=+1669.105401290" watchObservedRunningTime="2026-09-11 16:44:38.331701881 +0000 UTC m=+1669.106402460"
Sep 11 16:44:44.539942 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:44.539902    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:44:45.102302 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:45.102269    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:45.115149 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:45.115119    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:45.359068 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:45.358982    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:44:53.176172 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:44:53.176140    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:09.321619 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:09.321583    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:45:11.110696 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.110657    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:45:11.113295 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.113267    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:45:11.122318 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:45:11.122279    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:45:11.122318 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:45:11.122318 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:45:11.122318 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5" cmd=["/bin/sleep","15"]
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.122339    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" message="" err=<
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:45:11.122357    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main" containerID="cri-o://9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"
Sep 11 16:45:11.122617 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.122400    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main" containerID="cri-o://9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5" gracePeriod=60
Sep 11 16:45:11.175633 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.175591    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:11.230393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.230357    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:45:11.348833 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.348799    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Sep 11 16:45:11.466914 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.466891    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:45:11.471916 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.471890    2550 generic.go:356] "Generic (PLEG): container finished" podID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerID="9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5" exitCode=0
Sep 11 16:45:11.472062 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.471938    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"
Sep 11 16:45:11.472062 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.471979    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" event={"ID":"388f38ac-b453-47dd-84c9-c0a827f4c982","Type":"ContainerDied","Data":"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"}
Sep 11 16:45:11.472062 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.472032    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs" event={"ID":"388f38ac-b453-47dd-84c9-c0a827f4c982","Type":"ContainerDied","Data":"2e16de16d82d489d801255ebe4cc78bcc4792528723b769dc955d55eeafafffb"}
Sep 11 16:45:11.472062 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.472062    2550 scope.go:117] "RemoveContainer" containerID="9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"
Sep 11 16:45:11.481475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.481454    2550 scope.go:117] "RemoveContainer" containerID="9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"
Sep 11 16:45:11.481811 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:45:11.481789    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5\": container with ID starting with 9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5 not found: ID does not exist" containerID="9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"
Sep 11 16:45:11.481906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.481819    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5"} err="failed to get container status \"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5\": rpc error: code = NotFound desc = could not find container \"9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5\": container with ID starting with 9837e07cc2fc7f63527dcbb5801fe4186d6154f220e8311db6bb26f40053ffd5 not found: ID does not exist"
Sep 11 16:45:11.489889 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.489863    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.489916    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.489960    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490010    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490251 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490220    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qstr6\" (UniqueName: \"kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490251 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490233    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache" (OuterVolumeSpecName: "model-cache") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:11.490475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490349    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir\") pod \"388f38ac-b453-47dd-84c9-c0a827f4c982\" (UID: \"388f38ac-b453-47dd-84c9-c0a827f4c982\") "
Sep 11 16:45:11.490769 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490745    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home" (OuterVolumeSpecName: "home") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:11.490888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490866    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:11.491009 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.490993    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.491058 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.491013    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.491058 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.491025    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.492685 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.492646    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm" (OuterVolumeSpecName: "dshm") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:11.492685 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.492652    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6" (OuterVolumeSpecName: "kube-api-access-qstr6") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "kube-api-access-qstr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:45:11.492898 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.492878    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "388f38ac-b453-47dd-84c9-c0a827f4c982" (UID: "388f38ac-b453-47dd-84c9-c0a827f4c982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:45:11.591823 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.591786    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qstr6\" (UniqueName: \"kubernetes.io/projected/388f38ac-b453-47dd-84c9-c0a827f4c982-kube-api-access-qstr6\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.591823 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.591814    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388f38ac-b453-47dd-84c9-c0a827f4c982-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.591823 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.591826    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388f38ac-b453-47dd-84c9-c0a827f4c982-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:11.788941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.788914    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:45:11.793163 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:11.793133    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-55lvs"]
Sep 11 16:45:13.686744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:13.686645    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" path="/var/lib/kubelet/pods/388f38ac-b453-47dd-84c9-c0a827f4c982/volumes"
Sep 11 16:45:19.321842 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:19.321799    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:22.327969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.327932    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 11 16:45:22.328282 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:45:22.327970    2550 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main" containerID="cri-o://4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"
Sep 11 16:45:22.328282 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.328026    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main" containerID="cri-o://4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c" gracePeriod=49
Sep 11 16:45:22.467469 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.467446    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks_62fcb2ec-b769-4edf-aa8f-9da2237afc8d/main/0.log"
Sep 11 16:45:22.469698 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.469682    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:45:22.528849 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.528818    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks_62fcb2ec-b769-4edf-aa8f-9da2237afc8d/main/0.log"
Sep 11 16:45:22.529030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.528860    2550 generic.go:356] "Generic (PLEG): container finished" podID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerID="4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c" exitCode=1
Sep 11 16:45:22.529030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.528908    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"
Sep 11 16:45:22.529030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.528949    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" event={"ID":"62fcb2ec-b769-4edf-aa8f-9da2237afc8d","Type":"ContainerDied","Data":"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"}
Sep 11 16:45:22.529030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.529000    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks" event={"ID":"62fcb2ec-b769-4edf-aa8f-9da2237afc8d","Type":"ContainerDied","Data":"7550ca4fc252b7d5647b596ef77a082a2487bbcc0fe1f017b1fc9ed8c2568d83"}
Sep 11 16:45:22.529030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.529022    2550 scope.go:117] "RemoveContainer" containerID="4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"
Sep 11 16:45:22.537885 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.537868    2550 scope.go:117] "RemoveContainer" containerID="4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"
Sep 11 16:45:22.538138 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:45:22.538121    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c\": container with ID starting with 4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c not found: ID does not exist" containerID="4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"
Sep 11 16:45:22.538191 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.538146    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c"} err="failed to get container status \"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c\": rpc error: code = NotFound desc = could not find container \"4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c\": container with ID starting with 4d3bd810b2176588471b82ec73561a71375c3583a3259d200b2647ab49d15b6c not found: ID does not exist"
Sep 11 16:45:22.594435 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.594349    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f5fx\" (UniqueName: \"kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx\") pod \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") "
Sep 11 16:45:22.594435 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.594393    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs\") pod \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\" (UID: \"62fcb2ec-b769-4edf-aa8f-9da2237afc8d\") "
Sep 11 16:45:22.596563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.596534    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx" (OuterVolumeSpecName: "kube-api-access-7f5fx") pod "62fcb2ec-b769-4edf-aa8f-9da2237afc8d" (UID: "62fcb2ec-b769-4edf-aa8f-9da2237afc8d"). InnerVolumeSpecName "kube-api-access-7f5fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:45:22.596563 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.596554    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62fcb2ec-b769-4edf-aa8f-9da2237afc8d" (UID: "62fcb2ec-b769-4edf-aa8f-9da2237afc8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:45:22.695544 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.695516    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5fx\" (UniqueName: \"kubernetes.io/projected/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-kube-api-access-7f5fx\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:22.695544 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.695541    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62fcb2ec-b769-4edf-aa8f-9da2237afc8d-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:45:22.850876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.850798    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:45:22.856924 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:22.856902    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5zx7ks"]
Sep 11 16:45:23.687071 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:45:23.687036    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" path="/var/lib/kubelet/pods/62fcb2ec-b769-4edf-aa8f-9da2237afc8d/volumes"
Sep 11 16:46:01.210196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:01.210160    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:01.308483 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:01.308447    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:07.519832 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:07.519792    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:08.781698 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.781663    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:08.782130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782038    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main"
Sep 11 16:46:08.782130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782058    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main"
Sep 11 16:46:08.782130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782106    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main"
Sep 11 16:46:08.782130 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782117    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main"
Sep 11 16:46:08.782325 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782220    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fcb2ec-b769-4edf-aa8f-9da2237afc8d" containerName="main"
Sep 11 16:46:08.782325 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.782233    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="388f38ac-b453-47dd-84c9-c0a827f4c982" containerName="main"
Sep 11 16:46:08.785253 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.785237    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.789715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.789633    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-l4txw\""
Sep 11 16:46:08.789715 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.789676    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 11 16:46:08.789950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.789681    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 11 16:46:08.789950 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.789735    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 11 16:46:08.796294 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.796267    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:08.809386 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.809356    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:08.889388 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889352    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.889597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889438    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdmw\" (UniqueName: \"kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.889597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889487    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.889597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889512    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.889597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889531    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.889597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.889553    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.981930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.981893    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:46:08.985337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.985312    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:08.988232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.988209    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-zjlz6\""
Sep 11 16:46:08.990845 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.990816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdmw\" (UniqueName: \"kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.990987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.990894    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.990987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.990926    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.990987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.990949    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.991139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.991044    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.991139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.991123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.991420 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.991390    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.991545 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.991471    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.991609 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.991566    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.993598 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.993578    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.994139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.994106    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:08.997512 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:08.997485    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:46:09.004765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.004743    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdmw\" (UniqueName: \"kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-d6sfs\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:09.012010 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.011984    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:09.092670 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.092541    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.092670 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.092605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.092887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.092673    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.092887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.092727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.097470 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.097436    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:09.193208 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193174    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.193377 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193217    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.193377 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193256    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.193377 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193280    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.193588 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193566    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.193655 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.193583    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.195878 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.195849    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.202704 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.202678    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.296829 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.296794    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.377559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.377535    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:09.379928 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:46:09.379902    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0610d3cc_bfb4_4a0b_bb83_90e495b03336.slice/crio-fc41cfdfb836a5c651abb325f491917d775c95ca58a46a64130cb7983f0568f0 WatchSource:0}: Error finding container fc41cfdfb836a5c651abb325f491917d775c95ca58a46a64130cb7983f0568f0: Status 404 returned error can't find the container with id fc41cfdfb836a5c651abb325f491917d775c95ca58a46a64130cb7983f0568f0
Sep 11 16:46:09.578756 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.578725    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:46:09.582988 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:46:09.582951    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812cf5f1_696a_491b_b3d9_aa6ea6cbc6e6.slice/crio-438ff1d9436310546d620f127b5cb98a096446dd8781ba04ba5ccb256ae79794 WatchSource:0}: Error finding container 438ff1d9436310546d620f127b5cb98a096446dd8781ba04ba5ccb256ae79794: Status 404 returned error can't find the container with id 438ff1d9436310546d620f127b5cb98a096446dd8781ba04ba5ccb256ae79794
Sep 11 16:46:09.760133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.760090    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" event={"ID":"0610d3cc-bfb4-4a0b-bb83-90e495b03336","Type":"ContainerStarted","Data":"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"}
Sep 11 16:46:09.760343 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.760177    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" event={"ID":"0610d3cc-bfb4-4a0b-bb83-90e495b03336","Type":"ContainerStarted","Data":"fc41cfdfb836a5c651abb325f491917d775c95ca58a46a64130cb7983f0568f0"}
Sep 11 16:46:09.761711 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.761676    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" event={"ID":"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6","Type":"ContainerStarted","Data":"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"}
Sep 11 16:46:09.761858 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.761729    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" event={"ID":"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6","Type":"ContainerStarted","Data":"438ff1d9436310546d620f127b5cb98a096446dd8781ba04ba5ccb256ae79794"}
Sep 11 16:46:09.761858 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.761852    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:09.787923 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.787876    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" podStartSLOduration=1.7878613840000002 podStartE2EDuration="1.787861384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:46:09.787029539 +0000 UTC m=+1760.561730119" watchObservedRunningTime="2026-09-11 16:46:09.787861384 +0000 UTC m=+1760.562561963"
Sep 11 16:46:09.804767 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:09.804714    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podStartSLOduration=1.804698197 podStartE2EDuration="1.804698197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:46:09.803851993 +0000 UTC m=+1760.578552575" watchObservedRunningTime="2026-09-11 16:46:09.804698197 +0000 UTC m=+1760.579398774"
Sep 11 16:46:10.098444 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:10.098390    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:12.815480 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:12.815446    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:12.877192 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:12.877160    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:19.098599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:19.098564    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:19.111106 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:19.111082    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:19.823323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:19.823286    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:31.660336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:31.660283    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:37.618827 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:37.618786    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:37.748961 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:37.748926    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:40.769223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:40.769192    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:46:42.130776 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.130737    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:42.133257 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.133226    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:46:42.142133 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:46:42.142091    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:46:42.142133 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:46:42.142133 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:46:42.142133 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee" cmd=["/bin/sleep","15"]
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.142156    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" message="" err=<
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:46:42.142173    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerName="main" containerID="cri-o://6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"
Sep 11 16:46:42.142434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.142223    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerName="main" containerID="cri-o://6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee" gracePeriod=60
Sep 11 16:46:42.377227 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.377199    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:42.479209 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.479176    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:46:42.492498 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.492478    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:42.573662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573614    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.573807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573670    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.573807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573695    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.573807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573717    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.573807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573805    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vdmw\" (UniqueName: \"kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.574035 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573854    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache\") pod \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\" (UID: \"0610d3cc-bfb4-4a0b-bb83-90e495b03336\") "
Sep 11 16:46:42.574035 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.573870    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:42.574139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.574070    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home" (OuterVolumeSpecName: "home") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:42.574139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.574128    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.574322 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.574296    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache" (OuterVolumeSpecName: "model-cache") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:42.576027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.576004    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:46:42.576139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.576007    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm" (OuterVolumeSpecName: "dshm") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:42.576139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.576089    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw" (OuterVolumeSpecName: "kube-api-access-5vdmw") pod "0610d3cc-bfb4-4a0b-bb83-90e495b03336" (UID: "0610d3cc-bfb4-4a0b-bb83-90e495b03336"). InnerVolumeSpecName "kube-api-access-5vdmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:46:42.675397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.675339    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.675397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.675358    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0610d3cc-bfb4-4a0b-bb83-90e495b03336-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.675397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.675368    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.675397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.675376    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdmw\" (UniqueName: \"kubernetes.io/projected/0610d3cc-bfb4-4a0b-bb83-90e495b03336-kube-api-access-5vdmw\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.675397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.675385    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0610d3cc-bfb4-4a0b-bb83-90e495b03336-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:46:42.925189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.925156    2550 generic.go:356] "Generic (PLEG): container finished" podID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerID="6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee" exitCode=0
Sep 11 16:46:42.925189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.925192    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" event={"ID":"0610d3cc-bfb4-4a0b-bb83-90e495b03336","Type":"ContainerDied","Data":"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"}
Sep 11 16:46:42.925460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.925202    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"
Sep 11 16:46:42.925460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.925233    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs" event={"ID":"0610d3cc-bfb4-4a0b-bb83-90e495b03336","Type":"ContainerDied","Data":"fc41cfdfb836a5c651abb325f491917d775c95ca58a46a64130cb7983f0568f0"}
Sep 11 16:46:42.925460 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.925255    2550 scope.go:117] "RemoveContainer" containerID="6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"
Sep 11 16:46:42.934171 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.934148    2550 scope.go:117] "RemoveContainer" containerID="6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"
Sep 11 16:46:42.934432 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:46:42.934413    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee\": container with ID starting with 6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee not found: ID does not exist" containerID="6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"
Sep 11 16:46:42.934489 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.934440    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee"} err="failed to get container status \"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee\": rpc error: code = NotFound desc = could not find container \"6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee\": container with ID starting with 6f5bf92aea4bbb973d5a583823e1c3f9d02f8d78ea202b4ca28845bff781a1ee not found: ID does not exist"
Sep 11 16:46:42.946885 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.946859    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:42.950754 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:42.950731    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-d6sfs"]
Sep 11 16:46:43.686669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:43.686618    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" path="/var/lib/kubelet/pods/0610d3cc-bfb4-4a0b-bb83-90e495b03336/volumes"
Sep 11 16:46:50.768533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:50.768496    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:46:57.153947 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:46:57.153888    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main" containerID="cri-o://8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe" gracePeriod=45
Sep 11 16:47:00.769263 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:00.769224    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:10.768750 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:10.768712    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:10.769336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:10.768809    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:47:20.768963 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:20.768924    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:27.402084 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.402055    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:47:27.539553 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539527    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs\") pod \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") "
Sep 11 16:47:27.539746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539593    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6\") pod \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") "
Sep 11 16:47:27.539746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539652    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache\") pod \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") "
Sep 11 16:47:27.539746 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539675    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache\") pod \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\" (UID: \"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6\") "
Sep 11 16:47:27.539930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539878    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" (UID: "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:47:27.539988 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.539950    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" (UID: "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:47:27.541701 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.541675    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6" (OuterVolumeSpecName: "kube-api-access-q7hq6") pod "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" (UID: "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6"). InnerVolumeSpecName "kube-api-access-q7hq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:47:27.541808 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.541750    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" (UID: "812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:47:27.640440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.640414    2550 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tokenizer-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:47:27.640440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.640437    2550 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-cachi2-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:47:27.640603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.640447    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:47:27.640603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.640457    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hq6\" (UniqueName: \"kubernetes.io/projected/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6-kube-api-access-q7hq6\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:47:27.731580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.731533    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:27.831892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.831817    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:27.931823 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:27.931784    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:28.128431 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.128337    2550 generic.go:356] "Generic (PLEG): container finished" podID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerID="8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe" exitCode=0
Sep 11 16:47:28.128431 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.128378    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"
Sep 11 16:47:28.128431 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.128396    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" event={"ID":"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6","Type":"ContainerDied","Data":"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"}
Sep 11 16:47:28.128765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.128450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp" event={"ID":"812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6","Type":"ContainerDied","Data":"438ff1d9436310546d620f127b5cb98a096446dd8781ba04ba5ccb256ae79794"}
Sep 11 16:47:28.128765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.128479    2550 scope.go:117] "RemoveContainer" containerID="8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"
Sep 11 16:47:28.136724 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.136705    2550 scope.go:117] "RemoveContainer" containerID="8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"
Sep 11 16:47:28.136994 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:47:28.136977    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe\": container with ID starting with 8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe not found: ID does not exist" containerID="8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"
Sep 11 16:47:28.137045 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.137004    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe"} err="failed to get container status \"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe\": rpc error: code = NotFound desc = could not find container \"8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe\": container with ID starting with 8eadb137f91c4f89947ae19c341e6b5d2f594db4c17725819e387ce183b4aefe not found: ID does not exist"
Sep 11 16:47:28.148978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.148954    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:47:28.154802 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:28.154781    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcr29pp"]
Sep 11 16:47:29.693064 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:29.693032    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" path="/var/lib/kubelet/pods/812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6/volumes"
Sep 11 16:47:29.930378 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:29.930345    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:31.789497 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:31.789459    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:31.884077 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:31.884046    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:34.220882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:34.220845    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:37.877570 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:37.877538    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:39.843146 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:39.843106    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:39.904599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:39.904563    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:43.376680 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:43.376615    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:43.477418 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:43.477379    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:47:49.705836 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:47:49.705803    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:08.003558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.003520    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:48:08.004071 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004018    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerName="main"
Sep 11 16:48:08.004071 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004036    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerName="main"
Sep 11 16:48:08.004071 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004071    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main"
Sep 11 16:48:08.004255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004079    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main"
Sep 11 16:48:08.004255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004154    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="812cf5f1-696a-491b-b3d9-aa6ea6cbc6e6" containerName="main"
Sep 11 16:48:08.004255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.004167    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0610d3cc-bfb4-4a0b-bb83-90e495b03336" containerName="main"
Sep 11 16:48:08.007639 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.007610    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.011608 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.011584    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 11 16:48:08.011761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.011609    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 11 16:48:08.011761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.011611    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 11 16:48:08.011761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.011666    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6vttl\""
Sep 11 16:48:08.011761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.011584    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-9588j\""
Sep 11 16:48:08.019355 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.019331    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:48:08.053571 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.053546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:08.069962 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.069938    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5zz\" (UniqueName: \"kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.070055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.069973    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.070055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.069992    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.070055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.070022    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.070055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.070050    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.070221 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.070094    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171392 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171363    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5zz\" (UniqueName: \"kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171398    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171523 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171507    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171891 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171846    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171891 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171877    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.171962 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.171902    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.173780 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.173758    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.174037 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.174020    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.179323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.179297    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5zz\" (UniqueName: \"kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.254206 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.254125    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:48:08.319188 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.319160    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:08.592391 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.592368    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:48:08.594363 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:48:08.594327    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877cb1b0_f1d9_4d7b_a725_8883e02abdb3.slice/crio-f2805c50df5062b7aa97c5da4cb724ba5c1bab5203ffd0c3258a5107baf1ba84 WatchSource:0}: Error finding container f2805c50df5062b7aa97c5da4cb724ba5c1bab5203ffd0c3258a5107baf1ba84: Status 404 returned error can't find the container with id f2805c50df5062b7aa97c5da4cb724ba5c1bab5203ffd0c3258a5107baf1ba84
Sep 11 16:48:08.596371 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:08.596351    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:48:09.333127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:09.333090    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerStarted","Data":"f2805c50df5062b7aa97c5da4cb724ba5c1bab5203ffd0c3258a5107baf1ba84"}
Sep 11 16:48:11.344264 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:11.344228    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerStarted","Data":"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"}
Sep 11 16:48:11.344704 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:11.344308    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:12.350784 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:12.350695    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerStarted","Data":"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"}
Sep 11 16:48:12.370457 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:12.370408    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" podStartSLOduration=3.622445078 podStartE2EDuration="5.370393172s" totalImagesPullingTime="1.747948094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:12.368770458 +0000 UTC m=+1883.143471053" watchObservedRunningTime="2026-09-11 16:48:12.370393172 +0000 UTC m=+1883.145093793"
Sep 11 16:48:13.319722 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:13.319691    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:18.319989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:18.319950    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:18.332580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:18.332550    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:18.389017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:18.388984    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:29.395070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:29.395037    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:48:32.375737 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:48:32.375698    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:06.425651 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:49:06.425603    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:06.476046 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:49:06.476014    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:38.514129 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:49:38.514087    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:39.683237 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:49:39.683201    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:49:43.587670 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:49:43.587573    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:00.209320 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:00.209285    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:20.544190 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:20.544147    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:23.812027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:23.811992    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:25.803199 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.803152    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:25.808263 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.808234    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:50:25.817344 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:50:25.817314    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:50:25.817344 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.817344 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.817344 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0" cmd=["/bin/sleep","15"]
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.817358    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" message="" err=<
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:50:25.817369    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="main" containerID="cri-o://2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"
Sep 11 16:50:25.817558 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.817418    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="main" containerID="cri-o://2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0" gracePeriod=60
Sep 11 16:50:25.901705 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.901676    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:25.929014 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.928950    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="llm-d-routing-sidecar" containerID="cri-o://75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7" gracePeriod=60
Sep 11 16:50:25.973076 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.973046    2550 generic.go:356] "Generic (PLEG): container finished" podID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerID="2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0" exitCode=0
Sep 11 16:50:25.973239 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:25.973125    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerDied","Data":"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"}
Sep 11 16:50:26.180710 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.180684    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:50:26.312174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312142    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312178    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312226    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb5zz\" (UniqueName: \"kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312255    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312298    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312341    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache\") pod \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\" (UID: \"877cb1b0-f1d9-4d7b-a725-8883e02abdb3\") "
Sep 11 16:50:26.312599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312423    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home" (OuterVolumeSpecName: "home") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.312599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312499    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.312743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312647    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.312743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312703    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.312743 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.312724    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.314446 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.314417    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz" (OuterVolumeSpecName: "kube-api-access-zb5zz") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "kube-api-access-zb5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:26.314446 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.314416    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm" (OuterVolumeSpecName: "dshm") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.314646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.314594    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "877cb1b0-f1d9-4d7b-a725-8883e02abdb3" (UID: "877cb1b0-f1d9-4d7b-a725-8883e02abdb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:26.413474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.413389    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.413474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.413423    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.413474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.413440    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5zz\" (UniqueName: \"kubernetes.io/projected/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-kube-api-access-zb5zz\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.413474 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.413452    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877cb1b0-f1d9-4d7b-a725-8883e02abdb3-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.642343 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.642309    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:26.707861 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.707832    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:26.979819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.979731    2550 generic.go:356] "Generic (PLEG): container finished" podID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerID="75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7" exitCode=0
Sep 11 16:50:26.979819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.979797    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"
Sep 11 16:50:26.980340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.979820    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerDied","Data":"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"}
Sep 11 16:50:26.980340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.979878    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf" event={"ID":"877cb1b0-f1d9-4d7b-a725-8883e02abdb3","Type":"ContainerDied","Data":"f2805c50df5062b7aa97c5da4cb724ba5c1bab5203ffd0c3258a5107baf1ba84"}
Sep 11 16:50:26.980340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.979906    2550 scope.go:117] "RemoveContainer" containerID="2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"
Sep 11 16:50:26.989128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.989111    2550 scope.go:117] "RemoveContainer" containerID="75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"
Sep 11 16:50:26.996685 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.996668    2550 scope.go:117] "RemoveContainer" containerID="2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"
Sep 11 16:50:26.996916 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:50:26.996896    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0\": container with ID starting with 2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0 not found: ID does not exist" containerID="2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"
Sep 11 16:50:26.996971 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.996923    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0"} err="failed to get container status \"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0\": rpc error: code = NotFound desc = could not find container \"2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0\": container with ID starting with 2e6c63fcd9d35e864a9d66879ff5e57d1060df18285be8bcf47b56b4a6771bb0 not found: ID does not exist"
Sep 11 16:50:26.996971 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.996939    2550 scope.go:117] "RemoveContainer" containerID="75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"
Sep 11 16:50:26.997163 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:50:26.997146    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7\": container with ID starting with 75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7 not found: ID does not exist" containerID="75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"
Sep 11 16:50:26.997204 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:26.997166    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7"} err="failed to get container status \"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7\": rpc error: code = NotFound desc = could not find container \"75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7\": container with ID starting with 75aeb2be931eee3f57490ae030b4c1dfc5eeec56f33b9e2f172050afeaa698a7 not found: ID does not exist"
Sep 11 16:50:27.005262 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:27.005173    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:50:27.010784 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:27.010762    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-557kf"]
Sep 11 16:50:27.686377 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:27.686339    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" path="/var/lib/kubelet/pods/877cb1b0-f1d9-4d7b-a725-8883e02abdb3/volumes"
Sep 11 16:50:44.999650 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:44.999602    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:45.100980 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:45.100936    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:45.205751 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:45.205712    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:45.303127 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:45.303035    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:46.382174 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:46.382139    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:50.082157 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:50.082113    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:50:59.010866 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:50:59.010831    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:00.070550 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:00.070501    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:09.425292 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:09.425259    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:12.893108 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:12.893016    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:12.994307 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:12.994273    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:18.836828 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:18.836788    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:24.315115 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.315076    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:24.416676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.416642    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:24.627837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.627761    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:51:24.628197 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628179    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="llm-d-routing-sidecar"
Sep 11 16:51:24.628280 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628214    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="llm-d-routing-sidecar"
Sep 11 16:51:24.628280 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628239    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="main"
Sep 11 16:51:24.628280 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628248    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="main"
Sep 11 16:51:24.628436 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628342    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="main"
Sep 11 16:51:24.628436 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.628360    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="877cb1b0-f1d9-4d7b-a725-8883e02abdb3" containerName="llm-d-routing-sidecar"
Sep 11 16:51:24.630607 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.630585    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.635255 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.635229    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 11 16:51:24.635718 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.635698    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-7h7kp\""
Sep 11 16:51:24.635875 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.635702    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 11 16:51:24.635967 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.635706    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 11 16:51:24.636021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.635757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-57tm8\""
Sep 11 16:51:24.649651 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.649614    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:51:24.681557 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.681531    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:24.773770 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773741    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h66c\" (UniqueName: \"kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.773914 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773784    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.773914 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773891    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.774027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.774027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.774027 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.773967    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875163    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875199    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875225    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h66c\" (UniqueName: \"kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875257    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875333    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875359    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875684 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875664    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875704    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.875856 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.875830    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.877537 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.877514    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.877831 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.877815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.884117 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.884060    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h66c\" (UniqueName: \"kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.940991 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.940949    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:24.953000 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:24.952973    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:25.214853 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:25.214831    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:51:25.216663 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:51:25.216632    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a13a209_0586_4ad5_9b31_dedacf72b3e4.slice/crio-8f2bf699e39244824cf49c1501652d4c09771d7db76bb08228cdfe129f2ea3a9 WatchSource:0}: Error finding container 8f2bf699e39244824cf49c1501652d4c09771d7db76bb08228cdfe129f2ea3a9: Status 404 returned error can't find the container with id 8f2bf699e39244824cf49c1501652d4c09771d7db76bb08228cdfe129f2ea3a9
Sep 11 16:51:25.240753 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:25.240722    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerStarted","Data":"8f2bf699e39244824cf49c1501652d4c09771d7db76bb08228cdfe129f2ea3a9"}
Sep 11 16:51:26.246276 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:26.246244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerStarted","Data":"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"}
Sep 11 16:51:26.246805 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:26.246378    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:27.252699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:27.252656    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerStarted","Data":"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"}
Sep 11 16:51:27.273504 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:27.273454    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" podStartSLOduration=3.273433363 podStartE2EDuration="3.273433363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:27.27253572 +0000 UTC m=+2078.047236305" watchObservedRunningTime="2026-09-11 16:51:27.273433363 +0000 UTC m=+2078.048133942"
Sep 11 16:51:27.941295 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:27.941268    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:28.134403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:28.134370    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:30.357906 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:30.357827    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:30.458214 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:30.458188    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:33.821463 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:33.821426    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:33.887289 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:33.887257    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:34.941353 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:34.941317    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:34.953492 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:34.953469    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:35.301605 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:35.301577    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:35.302107 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:35.302085    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:51:43.764267 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:43.764228    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:50.969064 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:50.969027    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:51.070496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:51.070465    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:51.170599 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:51.170556    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:51.594583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:51.594546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:51.696750 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:51.696717    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:51.796987 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:51.796956    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:52.826060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:52.826020    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:53.784653 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:53.784600    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:56.041902 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:56.041872    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:51:57.441461 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:51:57.441429    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:03.118530 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:03.118496    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:03.176066 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:03.176025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:08.205684 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:08.205645    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:16.700209 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:16.700174    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:32.789757 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:32.789719    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:38.355008 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:38.354979    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:38.461997 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:38.456405    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:59.394275 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.394224    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:59.406589 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.406562    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:52:59.415108 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:52:59.415077    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:52:59.415108 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.415108 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.415108 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70" cmd=["/bin/sleep","15"]
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.415123    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" message="" err=<
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:52:59.415135    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="main" containerID="cri-o://f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"
Sep 11 16:52:59.415312 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.415176    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="main" containerID="cri-o://f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70" gracePeriod=60
Sep 11 16:52:59.491666 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.491635    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:52:59.678188 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.678101    2550 generic.go:356] "Generic (PLEG): container finished" podID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerID="f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70" exitCode=0
Sep 11 16:52:59.678326 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:52:59.678180    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerDied","Data":"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"}
Sep 11 16:53:00.026942 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.026910    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="llm-d-routing-sidecar" containerID="cri-o://2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b" gracePeriod=60
Sep 11 16:53:00.272203 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.272179    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:53:00.362966 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.362865    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.362966 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.362909    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h66c\" (UniqueName: \"kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.362966 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.362935    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.363281 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.362991    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.363281 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.363128    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.363281 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.363183    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache\") pod \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\" (UID: \"8a13a209-0586-4ad5-9b31-dedacf72b3e4\") "
Sep 11 16:53:00.363766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.363693    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.363889 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.363768    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home" (OuterVolumeSpecName: "home") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.363889 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.363801    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.365504 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.365472    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm" (OuterVolumeSpecName: "dshm") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.365825 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.365790    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c" (OuterVolumeSpecName: "kube-api-access-4h66c") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "kube-api-access-4h66c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:00.365979 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.365956    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a13a209-0586-4ad5-9b31-dedacf72b3e4" (UID: "8a13a209-0586-4ad5-9b31-dedacf72b3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:00.464434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464408    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.464434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464430    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.464434 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464439    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h66c\" (UniqueName: \"kubernetes.io/projected/8a13a209-0586-4ad5-9b31-dedacf72b3e4-kube-api-access-4h66c\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.464855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464448    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.464855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464456    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.464855 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.464464    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a13a209-0586-4ad5-9b31-dedacf72b3e4-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.684760 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.684685    2550 generic.go:356] "Generic (PLEG): container finished" podID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerID="2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b" exitCode=0
Sep 11 16:53:00.684760 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.684740    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"
Sep 11 16:53:00.684929 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.684775    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerDied","Data":"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"}
Sep 11 16:53:00.684929 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.684827    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc" event={"ID":"8a13a209-0586-4ad5-9b31-dedacf72b3e4","Type":"ContainerDied","Data":"8f2bf699e39244824cf49c1501652d4c09771d7db76bb08228cdfe129f2ea3a9"}
Sep 11 16:53:00.684929 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.684858    2550 scope.go:117] "RemoveContainer" containerID="f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"
Sep 11 16:53:00.694272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.694257    2550 scope.go:117] "RemoveContainer" containerID="2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"
Sep 11 16:53:00.701722 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.701704    2550 scope.go:117] "RemoveContainer" containerID="f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"
Sep 11 16:53:00.701985 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:53:00.701965    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70\": container with ID starting with f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70 not found: ID does not exist" containerID="f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"
Sep 11 16:53:00.702056 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.701990    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70"} err="failed to get container status \"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70\": rpc error: code = NotFound desc = could not find container \"f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70\": container with ID starting with f5282d6cf521ef58fc5bef547d30798afb5e157d328fe3b8154ff5ef1ff2db70 not found: ID does not exist"
Sep 11 16:53:00.702056 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.702005    2550 scope.go:117] "RemoveContainer" containerID="2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"
Sep 11 16:53:00.702256 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:53:00.702239    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b\": container with ID starting with 2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b not found: ID does not exist" containerID="2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"
Sep 11 16:53:00.702296 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.702263    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b"} err="failed to get container status \"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b\": rpc error: code = NotFound desc = could not find container \"2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b\": container with ID starting with 2829183f86836b2970043cd99d14d8b8b09dd32d00e94615734c65035b28860b not found: ID does not exist"
Sep 11 16:53:00.708332 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.708308    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:53:00.711962 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:00.711942    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-c2mgc"]
Sep 11 16:53:01.686930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:01.686896    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" path="/var/lib/kubelet/pods/8a13a209-0586-4ad5-9b31-dedacf72b3e4/volumes"
Sep 11 16:53:01.988259 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:01.988230    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:18.373930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:18.373888    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:36.356187 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:36.356151    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:39.982011 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:39.981975    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:40.082514 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:40.082479    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:40.184374 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:40.184332    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:45.055653 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:45.055606    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:47.807555 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:47.807518    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:48.052348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:48.052312    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:48.171055 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:48.170971    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:53.816737 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:53.816705    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:59.289475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:59.289428    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:53:59.362110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:53:59.362076    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:04.390688 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:04.390656    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:04.905876 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:04.905830    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:09.846714 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:09.846677    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:09.914982 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:09.914950    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:12.602861 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:12.602786    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:30.795229 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795198    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:54:30.795606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795531    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="llm-d-routing-sidecar"
Sep 11 16:54:30.795606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795547    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="llm-d-routing-sidecar"
Sep 11 16:54:30.795606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795572    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="main"
Sep 11 16:54:30.795606 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795583    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="main"
Sep 11 16:54:30.795857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795683    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="main"
Sep 11 16:54:30.795857 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.795701    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a13a209-0586-4ad5-9b31-dedacf72b3e4" containerName="llm-d-routing-sidecar"
Sep 11 16:54:30.798578 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.798561    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.801662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.801636    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-mzqfb\""
Sep 11 16:54:30.802669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.802599    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 11 16:54:30.802809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.802679    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:54:30.802809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.802695    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 11 16:54:30.802809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.802703    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 11 16:54:30.802809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.802605    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 11 16:54:30.813969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.813941    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:54:30.827021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.826995    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899435    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899507    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899648    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjm2\" (UniqueName: \"kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899712    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899750    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899833    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:30.904877 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:30.899862    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000545 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000509    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000551    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjm2\" (UniqueName: \"kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000581    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000611    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000692    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.000736 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000719    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001041 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001041 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.000787    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001150 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.001040    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001150 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.001053    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001150 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.001110    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001284 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.001190    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.001447 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.001425    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.003107 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.003087    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.003403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.003382    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.010413 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.010389    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjm2\" (UniqueName: \"kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.067315 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.067225    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:31.109580 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.109547    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:31.181810 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.181776    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:54:31.388750 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:31.388725    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:54:31.390495 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:54:31.390462    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080c2be5_8473_46fc_b7ba_58ce2ec3265f.slice/crio-ac0656dfe0de06ea192e6810816ce55ef79e027968f2b6133775fbfa3d311338 WatchSource:0}: Error finding container ac0656dfe0de06ea192e6810816ce55ef79e027968f2b6133775fbfa3d311338: Status 404 returned error can't find the container with id ac0656dfe0de06ea192e6810816ce55ef79e027968f2b6133775fbfa3d311338
Sep 11 16:54:32.086327 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:32.086295    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerStarted","Data":"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"}
Sep 11 16:54:32.086932 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:32.086346    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerStarted","Data":"ac0656dfe0de06ea192e6810816ce55ef79e027968f2b6133775fbfa3d311338"}
Sep 11 16:54:34.099334 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:34.099293    2550 generic.go:356] "Generic (PLEG): container finished" podID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerID="ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97" exitCode=0
Sep 11 16:54:34.099837 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:34.099371    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerDied","Data":"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"}
Sep 11 16:54:35.106196 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:35.106162    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerStarted","Data":"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"}
Sep 11 16:54:35.109826 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:35.109786    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:35.128050 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:35.128011    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" podStartSLOduration=2.950727107 podStartE2EDuration="5.127997655s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.177270548s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:35.126185725 +0000 UTC m=+2265.900886327" watchObservedRunningTime="2026-09-11 16:54:35.127997655 +0000 UTC m=+2265.902698272"
Sep 11 16:54:41.110352 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:41.110316    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:41.122611 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:41.122584    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:54:41.144548 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:54:41.144519    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:55:07.202347 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:07.202294    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:11.517030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:11.516987    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:27.557841 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:27.557796    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:27.663449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:27.663413    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:27.758810 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:27.758778    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:32.627290 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:32.627249    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:47.562902 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.562821    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:55:47.565918 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.565896    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.568495 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.568473    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 11 16:55:47.569556 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.569529    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 11 16:55:47.569727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.569601    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-pc99p\""
Sep 11 16:55:47.569727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.569661    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 11 16:55:47.575160 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.575134    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:55:47.598595 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.598566    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:55:47.635720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635695    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.635852 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635731    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.635852 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.635852 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635822    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.636025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.636025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.635928    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737046 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737011    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737080    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737097    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737146    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737232 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737184    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737718 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737555    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737727    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.737800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.737774    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.739699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.739676    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.739819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.739804    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.745482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.745463    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw\") pod \"leave-v1-kserve-65684d4f5-9w6fd\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:47.878284 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:47.878214    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:48.149338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:48.149304    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:55:48.151322 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:55:48.151290    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e150dd_c334_4cb4_acd6_85316cbf3633.slice/crio-92ad1ba0b911371e2b970b6de6f3aa7909ee0b406e92be588c01015c47c8566c WatchSource:0}: Error finding container 92ad1ba0b911371e2b970b6de6f3aa7909ee0b406e92be588c01015c47c8566c: Status 404 returned error can't find the container with id 92ad1ba0b911371e2b970b6de6f3aa7909ee0b406e92be588c01015c47c8566c
Sep 11 16:55:48.153423 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:48.153406    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:55:48.449888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:48.449853    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" event={"ID":"91e150dd-c334-4cb4-acd6-85316cbf3633","Type":"ContainerStarted","Data":"92ad1ba0b911371e2b970b6de6f3aa7909ee0b406e92be588c01015c47c8566c"}
Sep 11 16:55:49.457766 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:49.457718    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" event={"ID":"91e150dd-c334-4cb4-acd6-85316cbf3633","Type":"ContainerStarted","Data":"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"}
Sep 11 16:55:49.477253 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:49.477202    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" podStartSLOduration=1.318565791 podStartE2EDuration="2.477184841s" totalImagesPullingTime="1.15861905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:49.476459222 +0000 UTC m=+2340.251159816" watchObservedRunningTime="2026-09-11 16:55:49.477184841 +0000 UTC m=+2340.251885419"
Sep 11 16:55:49.878306 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:49.878274    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:57.878871 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:57.878833    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:57.890920 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:57.890891    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:55:58.513114 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:55:58.513083    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:56:09.543128 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:09.543055    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:15.091874 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.091837    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:15.164772 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.164737    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:15.551590 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.551559    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:56:15.555010 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.554991    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.557717 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.557694    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 11 16:56:15.565110 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.565085    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:56:15.574326 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.574302    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:15.665868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.665843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.666013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.665876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.666013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.665896    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.666013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.665930    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.666161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.666037    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.666161 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.666068    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdr6z\" (UniqueName: \"kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767090 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767050    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767285 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767122    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767293    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdr6z\" (UniqueName: \"kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767504 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767671 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767657    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767773 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767531    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767773 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.767973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.767944    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.769801 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.769779    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.770236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.770215    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.774957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.774936    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdr6z\" (UniqueName: \"kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z\") pod \"leave-v2-kserve-556b4dddf9-9xs4s\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:15.867188 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:15.867121    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:16.141155 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:16.141128    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:56:16.143182 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:56:16.143144    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9ba4d8_dfec_48e9_b894_f29785c652f9.slice/crio-09624304e595a3b7d7305dbe69a939e50fd4187658bc282543c84ebdfba926ba WatchSource:0}: Error finding container 09624304e595a3b7d7305dbe69a939e50fd4187658bc282543c84ebdfba926ba: Status 404 returned error can't find the container with id 09624304e595a3b7d7305dbe69a939e50fd4187658bc282543c84ebdfba926ba
Sep 11 16:56:16.585013 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:16.584976    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" event={"ID":"2f9ba4d8-dfec-48e9-b894-f29785c652f9","Type":"ContainerStarted","Data":"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"}
Sep 11 16:56:16.585186 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:16.585028    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" event={"ID":"2f9ba4d8-dfec-48e9-b894-f29785c652f9","Type":"ContainerStarted","Data":"09624304e595a3b7d7305dbe69a939e50fd4187658bc282543c84ebdfba926ba"}
Sep 11 16:56:16.602782 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:16.602734    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" podStartSLOduration=1.602717927 podStartE2EDuration="1.602717927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:16.601514023 +0000 UTC m=+2367.376214602" watchObservedRunningTime="2026-09-11 16:56:16.602717927 +0000 UTC m=+2367.377418505"
Sep 11 16:56:16.867818 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:16.867730    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:25.867882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:25.867847    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:25.880207 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:25.880178    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:26.643603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:26.643573    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:56:40.486937 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.486893    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:40.494912 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.494881    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:56:40.504762 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:56:40.504727    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:56:40.504762 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.504762 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.504762 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c" cmd=["/bin/sleep","15"]
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.504777    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" message="" err=<
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:56:40.504790    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="main" containerID="cri-o://47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"
Sep 11 16:56:40.505085 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.504836    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="main" containerID="cri-o://47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c" gracePeriod=60
Sep 11 16:56:40.582859 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.582820    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:40.849951 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.849926    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:56:40.986466 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986435    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986644 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986485    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986644 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986505    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986644 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986567    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986644 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986583    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjm2\" (UniqueName: \"kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986644 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986615    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986674    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986721    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location\") pod \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\" (UID: \"080c2be5-8473-46fc-b7ba-58ce2ec3265f\") "
Sep 11 16:56:40.986935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986754    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache" (OuterVolumeSpecName: "model-cache") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:40.986935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986783    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home" (OuterVolumeSpecName: "home") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:40.986935 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.986927    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:40.987213 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.987052    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:40.987213 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.987071    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:40.987213 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.987085    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:40.987213 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.987040    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:56:40.988808 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.988783    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:56:40.989209 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.989180    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm" (OuterVolumeSpecName: "dshm") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:40.989209 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:40.989183    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2" (OuterVolumeSpecName: "kube-api-access-wsjm2") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "kube-api-access-wsjm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:56:41.007515 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.007487    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "080c2be5-8473-46fc-b7ba-58ce2ec3265f" (UID: "080c2be5-8473-46fc-b7ba-58ce2ec3265f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.088336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.088262    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.088336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.088289    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjm2\" (UniqueName: \"kubernetes.io/projected/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kube-api-access-wsjm2\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.088336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.088299    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/080c2be5-8473-46fc-b7ba-58ce2ec3265f-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.088336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.088307    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/080c2be5-8473-46fc-b7ba-58ce2ec3265f-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.088336 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.088320    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080c2be5-8473-46fc-b7ba-58ce2ec3265f-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.713596 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.713558    2550 generic.go:356] "Generic (PLEG): container finished" podID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerID="47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c" exitCode=0
Sep 11 16:56:41.714134 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.713644    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"
Sep 11 16:56:41.714134 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.713660    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerDied","Data":"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"}
Sep 11 16:56:41.714134 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.713747    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5" event={"ID":"080c2be5-8473-46fc-b7ba-58ce2ec3265f","Type":"ContainerDied","Data":"ac0656dfe0de06ea192e6810816ce55ef79e027968f2b6133775fbfa3d311338"}
Sep 11 16:56:41.714134 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.713778    2550 scope.go:117] "RemoveContainer" containerID="47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"
Sep 11 16:56:41.723212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.723189    2550 scope.go:117] "RemoveContainer" containerID="ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"
Sep 11 16:56:41.733076 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.733058    2550 scope.go:117] "RemoveContainer" containerID="47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"
Sep 11 16:56:41.733354 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:56:41.733334    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c\": container with ID starting with 47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c not found: ID does not exist" containerID="47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"
Sep 11 16:56:41.733439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.733362    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c"} err="failed to get container status \"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c\": rpc error: code = NotFound desc = could not find container \"47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c\": container with ID starting with 47d61ebe13eb909b85054ee4bb5ecac8617de3f6ebd07d68d5f7b8be6761f62c not found: ID does not exist"
Sep 11 16:56:41.733439 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.733379    2550 scope.go:117] "RemoveContainer" containerID="ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"
Sep 11 16:56:41.733707 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:56:41.733685    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97\": container with ID starting with ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97 not found: ID does not exist" containerID="ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"
Sep 11 16:56:41.733811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.733712    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97"} err="failed to get container status \"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97\": rpc error: code = NotFound desc = could not find container \"ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97\": container with ID starting with ea8dac351b08c7e779413e7b399cf17c13d6e786cbe06d4919008e80553b3f97 not found: ID does not exist"
Sep 11 16:56:41.734943 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.734922    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:56:41.739223 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:41.739201    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-9m8f5"]
Sep 11 16:56:43.686791 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:43.686758    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" path="/var/lib/kubelet/pods/080c2be5-8473-46fc-b7ba-58ce2ec3265f/volumes"
Sep 11 16:56:45.487313 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:45.487284    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:45.587800 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:45.587766    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:45.688785 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:45.688753    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:45.765484 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:45.765397    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:56:45.796989 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:45.796951    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:46.069894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:46.069817    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:56:46.088064 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:46.088035    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:50.558811 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:50.558779    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:51.084316 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084284    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:56:51.084727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084708    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="storage-initializer"
Sep 11 16:56:51.084835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084730    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="storage-initializer"
Sep 11 16:56:51.084835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084774    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="main"
Sep 11 16:56:51.084835 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084783    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="main"
Sep 11 16:56:51.084986 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.084865    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="080c2be5-8473-46fc-b7ba-58ce2ec3265f" containerName="main"
Sep 11 16:56:51.089466 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.089444    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.092473 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.092451    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 11 16:56:51.092603 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.092453    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 11 16:56:51.093707 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.093691    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-jmlm4\""
Sep 11 16:56:51.093816 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.093692    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 11 16:56:51.097684 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.097659    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:56:51.099668 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.099641    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:56:51.177730 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177702    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.177869 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177740    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.177869 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177799    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.177869 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177834    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.177978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.177978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.177907    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz26w\" (UniqueName: \"kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.278832 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278804    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278842    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278900    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278919    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.278985    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz26w\" (UniqueName: \"kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279302 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.279282    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.279284    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.279398 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.279354    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.281241 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.281221    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.281557 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.281540    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.287381 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.287363    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz26w\" (UniqueName: \"kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w\") pod \"tri-v1-kserve-79f8f98bc5-m27dg\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.401216 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.401144    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:56:51.674210 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.674183    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:56:51.676304 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:56:51.676271    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b93e1e_9e02_4ab0_8ca8_c6382c365bc3.slice/crio-53ec594a5e37ca576efa50ebf0627c3327ec63fb2f269668cde345ee856aa20f WatchSource:0}: Error finding container 53ec594a5e37ca576efa50ebf0627c3327ec63fb2f269668cde345ee856aa20f: Status 404 returned error can't find the container with id 53ec594a5e37ca576efa50ebf0627c3327ec63fb2f269668cde345ee856aa20f
Sep 11 16:56:51.760969 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.760937    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" event={"ID":"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3","Type":"ContainerStarted","Data":"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"}
Sep 11 16:56:51.761091 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.760990    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" event={"ID":"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3","Type":"ContainerStarted","Data":"53ec594a5e37ca576efa50ebf0627c3327ec63fb2f269668cde345ee856aa20f"}
Sep 11 16:56:51.795273 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:51.795224    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" podStartSLOduration=0.795210239 podStartE2EDuration="795.210239ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:51.793523935 +0000 UTC m=+2402.568224712" watchObservedRunningTime="2026-09-11 16:56:51.795210239 +0000 UTC m=+2402.569910817"
Sep 11 16:56:52.401917 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:56:52.401882    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:57:00.784234 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:00.784186    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" podUID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" containerName="main" containerID="cri-o://2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d" gracePeriod=45
Sep 11 16:57:01.026323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.026298    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:57:01.063559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063486    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.063559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063540    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.063786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063598    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.063786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063677    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.063786 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063728    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.063958 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063801    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdr6z\" (UniqueName: \"kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z\") pod \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\" (UID: \"2f9ba4d8-dfec-48e9-b894-f29785c652f9\") "
Sep 11 16:57:01.064017 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.063940    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home" (OuterVolumeSpecName: "home") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.064075 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.064028    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.064133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.064109    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.064133 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.064127    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.064250 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.064180    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.065858 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.065834    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:01.066148 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.066128    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm" (OuterVolumeSpecName: "dshm") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.066212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.066157    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z" (OuterVolumeSpecName: "kube-api-access-kdr6z") pod "2f9ba4d8-dfec-48e9-b894-f29785c652f9" (UID: "2f9ba4d8-dfec-48e9-b894-f29785c652f9"). InnerVolumeSpecName "kube-api-access-kdr6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:01.088576 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.088529    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" podUID="91e150dd-c334-4cb4-acd6-85316cbf3633" containerName="main" containerID="cri-o://728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d" gracePeriod=45
Sep 11 16:57:01.165455 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.165431    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.165559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.165456    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdr6z\" (UniqueName: \"kubernetes.io/projected/2f9ba4d8-dfec-48e9-b894-f29785c652f9-kube-api-access-kdr6z\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.165559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.165467    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.165559 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.165496    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9ba4d8-dfec-48e9-b894-f29785c652f9-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.326482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.326457    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:57:01.366909 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.366877    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.366934    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367018    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367040    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367107    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367138    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache\") pod \"91e150dd-c334-4cb4-acd6-85316cbf3633\" (UID: \"91e150dd-c334-4cb4-acd6-85316cbf3633\") "
Sep 11 16:57:01.367236 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367212    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.367427 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367401    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.367505 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367423    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home" (OuterVolumeSpecName: "home") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.367505 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.367459    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache" (OuterVolumeSpecName: "model-cache") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.369079 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.369058    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:01.369172 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.369073    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw" (OuterVolumeSpecName: "kube-api-access-2kmhw") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "kube-api-access-2kmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:01.369344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.369327    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm" (OuterVolumeSpecName: "dshm") pod "91e150dd-c334-4cb4-acd6-85316cbf3633" (UID: "91e150dd-c334-4cb4-acd6-85316cbf3633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:01.402153 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.402129    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:57:01.414353 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.414329    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:57:01.468356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.468318    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.468356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.468355    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmhw\" (UniqueName: \"kubernetes.io/projected/91e150dd-c334-4cb4-acd6-85316cbf3633-kube-api-access-2kmhw\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.468356 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.468366    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e150dd-c334-4cb4-acd6-85316cbf3633-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.468600 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.468375    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.468600 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.468383    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e150dd-c334-4cb4-acd6-85316cbf3633-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:57:01.808224 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.808198    2550 generic.go:356] "Generic (PLEG): container finished" podID="91e150dd-c334-4cb4-acd6-85316cbf3633" containerID="728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d" exitCode=0
Sep 11 16:57:01.808641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.808242    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"
Sep 11 16:57:01.808641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.808272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" event={"ID":"91e150dd-c334-4cb4-acd6-85316cbf3633","Type":"ContainerDied","Data":"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"}
Sep 11 16:57:01.808641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.808329    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd" event={"ID":"91e150dd-c334-4cb4-acd6-85316cbf3633","Type":"ContainerDied","Data":"92ad1ba0b911371e2b970b6de6f3aa7909ee0b406e92be588c01015c47c8566c"}
Sep 11 16:57:01.808641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.808365    2550 scope.go:117] "RemoveContainer" containerID="728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"
Sep 11 16:57:01.809713 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.809691    2550 generic.go:356] "Generic (PLEG): container finished" podID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" containerID="2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d" exitCode=0
Sep 11 16:57:01.809775 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.809732    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"
Sep 11 16:57:01.809775 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.809761    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" event={"ID":"2f9ba4d8-dfec-48e9-b894-f29785c652f9","Type":"ContainerDied","Data":"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"}
Sep 11 16:57:01.809851 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.809800    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s" event={"ID":"2f9ba4d8-dfec-48e9-b894-f29785c652f9","Type":"ContainerDied","Data":"09624304e595a3b7d7305dbe69a939e50fd4187658bc282543c84ebdfba926ba"}
Sep 11 16:57:01.817344 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.817329    2550 scope.go:117] "RemoveContainer" containerID="728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"
Sep 11 16:57:01.817593 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:57:01.817574    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d\": container with ID starting with 728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d not found: ID does not exist" containerID="728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"
Sep 11 16:57:01.817676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.817604    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d"} err="failed to get container status \"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d\": rpc error: code = NotFound desc = could not find container \"728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d\": container with ID starting with 728d425132490bc9ad7033ed4b62ff7f1e8053c7fb8017cde1dad413284b883d not found: ID does not exist"
Sep 11 16:57:01.817676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.817636    2550 scope.go:117] "RemoveContainer" containerID="2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"
Sep 11 16:57:01.822277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.822260    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:57:01.825462 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.825444    2550 scope.go:117] "RemoveContainer" containerID="2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"
Sep 11 16:57:01.825733 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:57:01.825713    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d\": container with ID starting with 2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d not found: ID does not exist" containerID="2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"
Sep 11 16:57:01.825808 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.825739    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d"} err="failed to get container status \"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d\": rpc error: code = NotFound desc = could not find container \"2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d\": container with ID starting with 2ef414d2523b0665278657c97c129b0670e2b1c35557dd8279effa6e9dbf766d not found: ID does not exist"
Sep 11 16:57:01.826846 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.826825    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:57:01.830060 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.830036    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9xs4s"]
Sep 11 16:57:01.840570 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.840545    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:57:01.842902 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:01.842883    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-9w6fd"]
Sep 11 16:57:03.686400 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:03.686368    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" path="/var/lib/kubelet/pods/2f9ba4d8-dfec-48e9-b894-f29785c652f9/volumes"
Sep 11 16:57:03.686819 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:03.686776    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e150dd-c334-4cb4-acd6-85316cbf3633" path="/var/lib/kubelet/pods/91e150dd-c334-4cb4-acd6-85316cbf3633/volumes"
Sep 11 16:57:06.379102 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379065    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:57:06.379718 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379697    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e150dd-c334-4cb4-acd6-85316cbf3633" containerName="main"
Sep 11 16:57:06.379718 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379720    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e150dd-c334-4cb4-acd6-85316cbf3633" containerName="main"
Sep 11 16:57:06.379893 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379763    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" containerName="main"
Sep 11 16:57:06.379893 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379772    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" containerName="main"
Sep 11 16:57:06.379893 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379855    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e150dd-c334-4cb4-acd6-85316cbf3633" containerName="main"
Sep 11 16:57:06.379893 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.379867    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9ba4d8-dfec-48e9-b894-f29785c652f9" containerName="main"
Sep 11 16:57:06.384424 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.384406    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.387493 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.387473    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 11 16:57:06.393107 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.393085    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:57:06.407122 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.407095    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:06.513549 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513522    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhcq\" (UniqueName: \"kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.513690 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513552    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.513690 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513618    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.513781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513718    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.513781 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.513861 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.513801    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614441    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614475 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614475    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614754 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614653    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614754 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614728    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhcq\" (UniqueName: \"kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614869 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614869 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614803    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.614967 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614904    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.615008 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.614973    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.615109 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.615091    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.616716 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.616695    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.617246 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.617224    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.623907 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.623887    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhcq\" (UniqueName: \"kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq\") pod \"tri-v2-kserve-dcb554bf5-gx9h6\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.695965 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.695938    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:06.974882 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:06.974790    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:57:06.978116 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:57:06.978083    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ae1235_9e3a_47ac_925a_7ceaf44dfb60.slice/crio-7feeb3833acef6c0cb33ffd102133ce1875601d8c91ddb067f349dadcfa8b982 WatchSource:0}: Error finding container 7feeb3833acef6c0cb33ffd102133ce1875601d8c91ddb067f349dadcfa8b982: Status 404 returned error can't find the container with id 7feeb3833acef6c0cb33ffd102133ce1875601d8c91ddb067f349dadcfa8b982
Sep 11 16:57:07.846535 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:07.846474    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" event={"ID":"41ae1235-9e3a-47ac-925a-7ceaf44dfb60","Type":"ContainerStarted","Data":"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"}
Sep 11 16:57:07.846938 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:07.846540    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" event={"ID":"41ae1235-9e3a-47ac-925a-7ceaf44dfb60","Type":"ContainerStarted","Data":"7feeb3833acef6c0cb33ffd102133ce1875601d8c91ddb067f349dadcfa8b982"}
Sep 11 16:57:07.866025 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:07.865979    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" podStartSLOduration=1.865964043 podStartE2EDuration="1.865964043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:57:07.863935708 +0000 UTC m=+2418.638636286" watchObservedRunningTime="2026-09-11 16:57:07.865964043 +0000 UTC m=+2418.640664622"
Sep 11 16:57:08.696340 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:08.696308    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:14.271496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.271412    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:57:14.275034 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.275013    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.277645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.277602    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 11 16:57:14.285482 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.285460    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:57:14.380116 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380077    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.380272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.380272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380212    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.380272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380254    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.380387 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380327    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.380387 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.380361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgk7b\" (UniqueName: \"kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.386188 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.386165    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:14.481259 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481229    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481271    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481301    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgk7b\" (UniqueName: \"kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481615 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481432    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481615 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481481    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481777 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481753    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481868 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481849    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.481945 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.481928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.483662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.483637    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.484030 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.484005    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.489719 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.489697    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgk7b\" (UniqueName: \"kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b\") pod \"tri-v3-kserve-798c88c66c-xkmcl\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.585918 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.585855    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:14.858348 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.858271    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:57:14.861522 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:57:14.861495    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649e4daf_a828_4b8f_a899_0043b6ffb2c6.slice/crio-8b1792eeb3b3f76beadd85115b7292853b97cb56af4c42d32e25ca2735e7cebb WatchSource:0}: Error finding container 8b1792eeb3b3f76beadd85115b7292853b97cb56af4c42d32e25ca2735e7cebb: Status 404 returned error can't find the container with id 8b1792eeb3b3f76beadd85115b7292853b97cb56af4c42d32e25ca2735e7cebb
Sep 11 16:57:14.880676 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:14.880598    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" event={"ID":"649e4daf-a828-4b8f-a899-0043b6ffb2c6","Type":"ContainerStarted","Data":"8b1792eeb3b3f76beadd85115b7292853b97cb56af4c42d32e25ca2735e7cebb"}
Sep 11 16:57:15.887613 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:15.887564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" event={"ID":"649e4daf-a828-4b8f-a899-0043b6ffb2c6","Type":"ContainerStarted","Data":"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"}
Sep 11 16:57:15.904993 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:15.904945    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" podStartSLOduration=1.9049287480000001 podStartE2EDuration="1.904928748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:57:15.904612973 +0000 UTC m=+2426.679313554" watchObservedRunningTime="2026-09-11 16:57:15.904928748 +0000 UTC m=+2426.679629326"
Sep 11 16:57:16.585984 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:16.585949    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:16.697140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:16.697108    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:16.709699 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:16.709668    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:16.903553 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:16.903467    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:57:24.586364 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:24.586325    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:24.598774 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:24.598751    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:24.941021 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:24.940943    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:57:27.626922 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:27.626886    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:29.286063 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:29.286025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:29.383549 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:29.383513    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:33.199541 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:33.199499    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:33.260058 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:33.260023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:35.063964 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:35.063926    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:40.756597 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:40.756562    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:40.821992 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:40.821953    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:54.187299 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:54.187257    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:54.289663 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:54.289616    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:59.296888 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.296857    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:57:59.300342 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.300320    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.303386 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.303362    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 11 16:57:59.303539 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.303467    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 11 16:57:59.304708 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.304687    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 11 16:57:59.304708 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.304694    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:57:59.304896 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.304696    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 11 16:57:59.304896 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.304716    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-bqx9c\""
Sep 11 16:57:59.313694 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.313674    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:57:59.336688 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.336660    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:59.357682 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357656    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.357807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357693    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.357807 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357771    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.357894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.357894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357863    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.357894 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.358007 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw6j\" (UniqueName: \"kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.358007 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.357982    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458585 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw6j\" (UniqueName: \"kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458600    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458650    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458667    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458703    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458727 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458999 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458750    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.458999 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.458769    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.459144 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.459118    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.459272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.459234    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.459385 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.459347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.459452 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.459405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.459848 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.459828    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.461440 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.461409    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.461549 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.461441    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.468637 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.468600    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw6j\" (UniqueName: \"kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-vns98\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.612303 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.612216    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:57:59.644465 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.644426    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:59.739359 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.739322    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:57:59.888700 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:57:59.888665    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:57:59.890817 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:57:59.890785    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4d102_6006_4a24_8ba0_55a35e0989fe.slice/crio-cc4eb9f42f28067d76dbbdc36be03c1b5b48f0abba9fe0d840f700f25145a680 WatchSource:0}: Error finding container cc4eb9f42f28067d76dbbdc36be03c1b5b48f0abba9fe0d840f700f25145a680: Status 404 returned error can't find the container with id cc4eb9f42f28067d76dbbdc36be03c1b5b48f0abba9fe0d840f700f25145a680
Sep 11 16:58:00.096301 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:00.096261    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerStarted","Data":"df29e28886cc42e6057a6a0f93e99c6b8c6b205765d9f02fd8f7873c3f41dba2"}
Sep 11 16:58:00.096477 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:00.096324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerStarted","Data":"cc4eb9f42f28067d76dbbdc36be03c1b5b48f0abba9fe0d840f700f25145a680"}
Sep 11 16:58:03.112691 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:03.112655    2550 generic.go:356] "Generic (PLEG): container finished" podID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerID="df29e28886cc42e6057a6a0f93e99c6b8c6b205765d9f02fd8f7873c3f41dba2" exitCode=0
Sep 11 16:58:03.113139 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:03.112725    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerDied","Data":"df29e28886cc42e6057a6a0f93e99c6b8c6b205765d9f02fd8f7873c3f41dba2"}
Sep 11 16:58:04.119069 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:04.119034    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerStarted","Data":"b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75"}
Sep 11 16:58:04.139975 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:04.139934    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" podStartSLOduration=2.856375002 podStartE2EDuration="5.139921425s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.283546423s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:04.138003196 +0000 UTC m=+2474.912703775" watchObservedRunningTime="2026-09-11 16:58:04.139921425 +0000 UTC m=+2474.914622002"
Sep 11 16:58:04.612744 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:04.612710    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:09.612508 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:09.612473    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:09.624657 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:09.624611    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:10.156412 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:10.156380    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:10.576646 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:10.576580    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:25.525292 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:25.525250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:25.627649 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:25.627589    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:25.690183 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:25.690153    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:58:25.712361 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:25.712324    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:25.772258 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:25.772233    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:26.372547 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:26.372511    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:58:26.391313 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:26.391284    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:26.963244 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:26.963206    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:58:26.986067 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:26.986026    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:30.608018 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:30.607982    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:32.482117 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:32.482083    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:32.582304 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:32.582270    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:36.747520 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:36.747484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:40.710077 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:40.710040    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerName="main" containerID="cri-o://4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887" gracePeriod=45
Sep 11 16:58:40.961200 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:40.961145    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:58:41.034840 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034813    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.034983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034860    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.034983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034884    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.034983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034903    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.034983 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034973    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.035208 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.034997    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgk7b\" (UniqueName: \"kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b\") pod \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\" (UID: \"649e4daf-a828-4b8f-a899-0043b6ffb2c6\") "
Sep 11 16:58:41.035208 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035076    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.035208 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035155    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.035208 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035196    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home" (OuterVolumeSpecName: "home") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.035393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035322    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.035393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035334    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.035393 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.035343    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.037219 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.037197    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:41.037319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.037232    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm" (OuterVolumeSpecName: "dshm") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.037319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.037248    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b" (OuterVolumeSpecName: "kube-api-access-cgk7b") pod "649e4daf-a828-4b8f-a899-0043b6ffb2c6" (UID: "649e4daf-a828-4b8f-a899-0043b6ffb2c6"). InnerVolumeSpecName "kube-api-access-cgk7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:41.136297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.136267    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649e4daf-a828-4b8f-a899-0043b6ffb2c6-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.136297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.136293    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e4daf-a828-4b8f-a899-0043b6ffb2c6-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.136297 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.136305    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgk7b\" (UniqueName: \"kubernetes.io/projected/649e4daf-a828-4b8f-a899-0043b6ffb2c6-kube-api-access-cgk7b\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.293323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.293240    2550 generic.go:356] "Generic (PLEG): container finished" podID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerID="4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887" exitCode=0
Sep 11 16:58:41.293323 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.293288    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"
Sep 11 16:58:41.293520 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.293320    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" event={"ID":"649e4daf-a828-4b8f-a899-0043b6ffb2c6","Type":"ContainerDied","Data":"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"}
Sep 11 16:58:41.293520 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.293374    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" event={"ID":"649e4daf-a828-4b8f-a899-0043b6ffb2c6","Type":"ContainerDied","Data":"8b1792eeb3b3f76beadd85115b7292853b97cb56af4c42d32e25ca2735e7cebb"}
Sep 11 16:58:41.293520 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.293406    2550 scope.go:117] "RemoveContainer" containerID="4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"
Sep 11 16:58:41.302965 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.302950    2550 scope.go:117] "RemoveContainer" containerID="4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"
Sep 11 16:58:41.303216 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:58:41.303200    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887\": container with ID starting with 4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887 not found: ID does not exist" containerID="4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"
Sep 11 16:58:41.303262 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.303224    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887"} err="failed to get container status \"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887\": rpc error: code = NotFound desc = could not find container \"4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887\": container with ID starting with 4f602d5205d2cca9e1353d68bb4336220b654bdb455a17e463c644868af6a887 not found: ID does not exist"
Sep 11 16:58:41.316641 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.316590    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:58:41.319930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.319909    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl"]
Sep 11 16:58:41.390957 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.390920    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" podUID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" containerName="main" containerID="cri-o://2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3" gracePeriod=45
Sep 11 16:58:41.630064 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.630042    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:58:41.687140 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.687110    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" path="/var/lib/kubelet/pods/649e4daf-a828-4b8f-a899-0043b6ffb2c6/volumes"
Sep 11 16:58:41.740679 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740653    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740693    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740710    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740791    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740814    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz26w\" (UniqueName: \"kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w\") pod \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\" (UID: \"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3\") "
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.740962    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.741056    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home" (OuterVolumeSpecName: "home") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.741070 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.741066    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.742887 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.742861    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:41.743005 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.742901    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w" (OuterVolumeSpecName: "kube-api-access-sz26w") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "kube-api-access-sz26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:41.743005 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.742905    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm" (OuterVolumeSpecName: "dshm") pod "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" (UID: "25b93e1e-9e02-4ab0-8ca8-c6382c365bc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:41.841583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841513    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.841583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841560    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.841583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841571    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz26w\" (UniqueName: \"kubernetes.io/projected/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-kube-api-access-sz26w\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.841583 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841580    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.841852 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841589    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.841852 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.841596    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:41.930941 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.930902    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-xkmcl" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded"
Sep 11 16:58:41.982272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:41.982235    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" podUID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" containerName="main" containerID="cri-o://e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004" gracePeriod=45
Sep 11 16:58:42.224817 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.224793    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:58:42.300102 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.300018    2550 generic.go:356] "Generic (PLEG): container finished" podID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" containerID="e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004" exitCode=0
Sep 11 16:58:42.300102 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.300065    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"
Sep 11 16:58:42.300337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.300098    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" event={"ID":"41ae1235-9e3a-47ac-925a-7ceaf44dfb60","Type":"ContainerDied","Data":"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"}
Sep 11 16:58:42.300337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.300149    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6" event={"ID":"41ae1235-9e3a-47ac-925a-7ceaf44dfb60","Type":"ContainerDied","Data":"7feeb3833acef6c0cb33ffd102133ce1875601d8c91ddb067f349dadcfa8b982"}
Sep 11 16:58:42.300337 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.300171    2550 scope.go:117] "RemoveContainer" containerID="e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"
Sep 11 16:58:42.302282 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.302254    2550 generic.go:356] "Generic (PLEG): container finished" podID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" containerID="2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3" exitCode=0
Sep 11 16:58:42.302414 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.302307    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"
Sep 11 16:58:42.302414 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.302335    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" event={"ID":"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3","Type":"ContainerDied","Data":"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"}
Sep 11 16:58:42.302414 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.302379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg" event={"ID":"25b93e1e-9e02-4ab0-8ca8-c6382c365bc3","Type":"ContainerDied","Data":"53ec594a5e37ca576efa50ebf0627c3327ec63fb2f269668cde345ee856aa20f"}
Sep 11 16:58:42.308993 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.308974    2550 scope.go:117] "RemoveContainer" containerID="e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"
Sep 11 16:58:42.309259 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:58:42.309236    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004\": container with ID starting with e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004 not found: ID does not exist" containerID="e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"
Sep 11 16:58:42.309354 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.309265    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004"} err="failed to get container status \"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004\": rpc error: code = NotFound desc = could not find container \"e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004\": container with ID starting with e2aade2fa264c439a271290401c5276db9555c319a3df99df75fb4e6db37e004 not found: ID does not exist"
Sep 11 16:58:42.309354 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.309281    2550 scope.go:117] "RemoveContainer" containerID="2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"
Sep 11 16:58:42.317403 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.317379    2550 scope.go:117] "RemoveContainer" containerID="2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"
Sep 11 16:58:42.317671 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:58:42.317652    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3\": container with ID starting with 2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3 not found: ID does not exist" containerID="2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"
Sep 11 16:58:42.317734 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.317676    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3"} err="failed to get container status \"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3\": rpc error: code = NotFound desc = could not find container \"2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3\": container with ID starting with 2b4274ec3837652148104c208e01b45ae0e4fb5427b12b04acaa0267057ff1e3 not found: ID does not exist"
Sep 11 16:58:42.323998 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.323978    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:58:42.328003 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.327983    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-m27dg"]
Sep 11 16:58:42.345411 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345391    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345527 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345422    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345527 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345476    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhhcq\" (UniqueName: \"kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345530    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345662 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345650    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345763 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345679    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir\") pod \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\" (UID: \"41ae1235-9e3a-47ac-925a-7ceaf44dfb60\") "
Sep 11 16:58:42.345763 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache" (OuterVolumeSpecName: "model-cache") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:42.345930 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345907    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home" (OuterVolumeSpecName: "home") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:42.346004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345971    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:42.346004 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.345982    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.347645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.347564    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:42.347645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.347580    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq" (OuterVolumeSpecName: "kube-api-access-jhhcq") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "kube-api-access-jhhcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:42.347645 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.347600    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm" (OuterVolumeSpecName: "dshm") pod "41ae1235-9e3a-47ac-925a-7ceaf44dfb60" (UID: "41ae1235-9e3a-47ac-925a-7ceaf44dfb60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:42.446755 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.446729    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.446755 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.446754    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.446892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.446767    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.446892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.446781    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.446892 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.446794    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhcq\" (UniqueName: \"kubernetes.io/projected/41ae1235-9e3a-47ac-925a-7ceaf44dfb60-kube-api-access-jhhcq\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:42.622669 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.622634    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:58:42.626259 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.626239    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gx9h6"]
Sep 11 16:58:42.813660 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.813605    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:58:42.818525 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.818496    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:42.823864 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:58:42.823837    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:58:42.823864 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.823864 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.823864 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75" cmd=["/bin/sleep","15"]
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.823880    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" message="" err=<
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]: E0911 16:58:42.823893    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main" containerID="cri-o://b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75"
Sep 11 16:58:42.824100 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.823926    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main" containerID="cri-o://b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75" gracePeriod=60
Sep 11 16:58:42.878721 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:42.878610    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:58:43.145844 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.145752    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 11 16:58:43.311371 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.311341    2550 generic.go:356] "Generic (PLEG): container finished" podID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerID="b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75" exitCode=0
Sep 11 16:58:43.311538 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.311377    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerDied","Data":"b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75"}
Sep 11 16:58:43.566424 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.566392    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:43.658448 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658412    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658466    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658493    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658517    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrw6j\" (UniqueName: \"kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658537    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658664 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658563    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658730    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.658952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658813    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.658952 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658869    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir\") pod \"52d4d102-6006-4a24-8ba0-55a35e0989fe\" (UID: \"52d4d102-6006-4a24-8ba0-55a35e0989fe\") "
Sep 11 16:58:43.659105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.658983    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:58:43.659105 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.659066    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home" (OuterVolumeSpecName: "home") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.659226 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.659203    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.659287 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.659228    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.659287 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.659244    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d4d102-6006-4a24-8ba0-55a35e0989fe-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.659287 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.659257    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.660862 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.660841    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:43.661168 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.661144    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j" (OuterVolumeSpecName: "kube-api-access-lrw6j") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "kube-api-access-lrw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:43.661168 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.661141    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm" (OuterVolumeSpecName: "dshm") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.678867 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.678844    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52d4d102-6006-4a24-8ba0-55a35e0989fe" (UID: "52d4d102-6006-4a24-8ba0-55a35e0989fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.687087 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.687030    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" path="/var/lib/kubelet/pods/25b93e1e-9e02-4ab0-8ca8-c6382c365bc3/volumes"
Sep 11 16:58:43.687425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.687412    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" path="/var/lib/kubelet/pods/41ae1235-9e3a-47ac-925a-7ceaf44dfb60/volumes"
Sep 11 16:58:43.760168 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.760139    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4d102-6006-4a24-8ba0-55a35e0989fe-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.760168 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.760167    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw6j\" (UniqueName: \"kubernetes.io/projected/52d4d102-6006-4a24-8ba0-55a35e0989fe-kube-api-access-lrw6j\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.760338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.760178    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.760338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.760187    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.760338 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:43.760195    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d4d102-6006-4a24-8ba0-55a35e0989fe-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 16:58:44.317660 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.317609    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98" event={"ID":"52d4d102-6006-4a24-8ba0-55a35e0989fe","Type":"ContainerDied","Data":"cc4eb9f42f28067d76dbbdc36be03c1b5b48f0abba9fe0d840f700f25145a680"}
Sep 11 16:58:44.317660 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.317648    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"
Sep 11 16:58:44.318202 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.317683    2550 scope.go:117] "RemoveContainer" containerID="b3a325c39798e42763d4c12956efd2bf1e957f1c9a4454b3609b6a7287e93a75"
Sep 11 16:58:44.326697 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.326682    2550 scope.go:117] "RemoveContainer" containerID="df29e28886cc42e6057a6a0f93e99c6b8c6b205765d9f02fd8f7873c3f41dba2"
Sep 11 16:58:44.337592 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.337565    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:58:44.342551 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:44.342526    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-vns98"]
Sep 11 16:58:45.686118 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:58:45.686084    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" path="/var/lib/kubelet/pods/52d4d102-6006-4a24-8ba0-55a35e0989fe/volumes"
Sep 11 16:59:04.425039 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:04.425000    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:07.081490 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.081453    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 16:59:07.081978 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.081961    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" containerName="main"
Sep 11 16:59:07.082036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.081981    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" containerName="main"
Sep 11 16:59:07.082036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082000    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="storage-initializer"
Sep 11 16:59:07.082036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082009    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="storage-initializer"
Sep 11 16:59:07.082036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082025    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" containerName="main"
Sep 11 16:59:07.082036 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082034    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082051    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082060    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082079    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082088    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082170    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ae1235-9e3a-47ac-925a-7ceaf44dfb60" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082180    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d4d102-6006-4a24-8ba0-55a35e0989fe" containerName="main"
Sep 11 16:59:07.082189 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082189    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="649e4daf-a828-4b8f-a899-0043b6ffb2c6" containerName="main"
Sep 11 16:59:07.082396 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.082202    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b93e1e-9e02-4ab0-8ca8-c6382c365bc3" containerName="main"
Sep 11 16:59:07.085028 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.085007    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.087761 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.087740    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 11 16:59:07.089221 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.089197    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-6tbhd\""
Sep 11 16:59:07.089392 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.089367    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 11 16:59:07.089480 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.089428    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 11 16:59:07.096277 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.096254    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 16:59:07.115780 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.115752    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:07.257449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.257449 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257449    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.257692 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257509    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.257692 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257533    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdsg\" (UniqueName: \"kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.257692 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.257808 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.257704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdsg\" (UniqueName: \"kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358422    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358451 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358449    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358470    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358484    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358720 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358526    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358925 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358901    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.358967 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358917    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.359019 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.358963    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.360779 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.360750    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.361106 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.361091    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.366809 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.366791    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdsg\" (UniqueName: \"kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg\") pod \"late-v2-kserve-86766546f-q9qgw\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.398694 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.398668    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:07.668701 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:07.668603    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 16:59:07.672670 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:59:07.672641    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bfaeef_0038_4524_b1be_48457f0695d1.slice/crio-65f341102149e72d7af8f4e927adfe73f4a157cbac2c446591cbbdd9ce84cca9 WatchSource:0}: Error finding container 65f341102149e72d7af8f4e927adfe73f4a157cbac2c446591cbbdd9ce84cca9: Status 404 returned error can't find the container with id 65f341102149e72d7af8f4e927adfe73f4a157cbac2c446591cbbdd9ce84cca9
Sep 11 16:59:08.423962 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:08.423924    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" event={"ID":"c8bfaeef-0038-4524-b1be-48457f0695d1","Type":"ContainerStarted","Data":"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"}
Sep 11 16:59:08.424420 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:08.423986    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" event={"ID":"c8bfaeef-0038-4524-b1be-48457f0695d1","Type":"ContainerStarted","Data":"65f341102149e72d7af8f4e927adfe73f4a157cbac2c446591cbbdd9ce84cca9"}
Sep 11 16:59:08.443675 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:08.443602    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" podStartSLOduration=1.443585393 podStartE2EDuration="1.443585393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:08.441474367 +0000 UTC m=+2539.216174942" watchObservedRunningTime="2026-09-11 16:59:08.443585393 +0000 UTC m=+2539.218285970"
Sep 11 16:59:09.399059 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:09.399018    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:10.235839 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:10.235804    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:10.298496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:10.298458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:17.399144 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:17.399100    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:17.411765 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:17.411737    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:17.474109 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:17.474081    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 16:59:29.767963 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:29.767926    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:29.865212 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:29.865175    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:35.838319 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:35.838285    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:36.396364 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.396330    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 16:59:36.400315 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.400285    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.403201 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.403177    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 11 16:59:36.403327 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.403224    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 11 16:59:36.403327 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.403224    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:59:36.404278 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.404257    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 11 16:59:36.404378 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.404366    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 11 16:59:36.404441 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.404413    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-jcvlw\""
Sep 11 16:59:36.410594 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.410572    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 16:59:36.430612 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.430587    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:36.506272 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506281    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506397 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506407    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5fv\" (UniqueName: \"kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506516 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506505    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506534    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.506661 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.506582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607253 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607224    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607270    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607295    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5fv\" (UniqueName: \"kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607313    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607346    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607375    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607425 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607843 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607439    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607843 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607715    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.607953 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.607909    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.608173 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.608150    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.608238 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.608180    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.608330 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.608310    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.609823 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.609797    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.610350 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.610331    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.618973 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.618953    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5fv\" (UniqueName: \"kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.657654 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.657558    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:36.712391 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.712362    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:36.719773 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.719735    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:36.990092 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:36.990064    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 16:59:36.991726 ip-10-0-135-175 kubenswrapper[2550]: W0911 16:59:36.991701    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0147e42_ec2c_4f1f_addf_a678d3bfdb35.slice/crio-b3f15e0d658ba2d267105a473e2e9f729c8ef2958a4e05bf6b66387f7a8bfe67 WatchSource:0}: Error finding container b3f15e0d658ba2d267105a473e2e9f729c8ef2958a4e05bf6b66387f7a8bfe67: Status 404 returned error can't find the container with id b3f15e0d658ba2d267105a473e2e9f729c8ef2958a4e05bf6b66387f7a8bfe67
Sep 11 16:59:37.564533 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:37.564498    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerStarted","Data":"947e5a8c26872874a785b2e6fc02327afeb923c4c5ef9a7a6f7d0b3b18cf932b"}
Sep 11 16:59:37.564804 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:37.564548    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerStarted","Data":"b3f15e0d658ba2d267105a473e2e9f729c8ef2958a4e05bf6b66387f7a8bfe67"}
Sep 11 16:59:39.575924 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:39.575838    2550 generic.go:356] "Generic (PLEG): container finished" podID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerID="947e5a8c26872874a785b2e6fc02327afeb923c4c5ef9a7a6f7d0b3b18cf932b" exitCode=0
Sep 11 16:59:39.575924 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:39.575909    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerDied","Data":"947e5a8c26872874a785b2e6fc02327afeb923c4c5ef9a7a6f7d0b3b18cf932b"}
Sep 11 16:59:40.581942 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:40.581911    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerStarted","Data":"0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb"}
Sep 11 16:59:40.602665 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:40.602601    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" podStartSLOduration=2.418719794 podStartE2EDuration="4.602586451s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.183866657s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:40.601995071 +0000 UTC m=+2571.376695650" watchObservedRunningTime="2026-09-11 16:59:40.602586451 +0000 UTC m=+2571.377287031"
Sep 11 16:59:40.712446 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:40.712417    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:41.435996 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:41.435957    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:41.536614 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:41.536559    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:46.713370 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:46.713328    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:46.725784 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:46.725761    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:47.626197 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:47.626167    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 16:59:52.022848 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:52.022807    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:52.132782 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:52.132745    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:52.234918 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:52.234889    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:54.098496 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:54.098457    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:56.297457 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:56.297414    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:57.001891 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:57.001853    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 16:59:57.098028 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:57.097986    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 16:59:57.278178 ip-10-0-135-175 kubenswrapper[2550]: I0911 16:59:57.278091    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:12.021096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.021033    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" podUID="c8bfaeef-0038-4524-b1be-48457f0695d1" containerName="main" containerID="cri-o://71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1" gracePeriod=45
Sep 11 17:00:12.263380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.263327    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 17:00:12.310023 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.309946    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310023 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.309988    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chdsg\" (UniqueName: \"kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310023 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310008    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310296 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310063    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310296 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310148    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310296 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310180    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir\") pod \"c8bfaeef-0038-4524-b1be-48457f0695d1\" (UID: \"c8bfaeef-0038-4524-b1be-48457f0695d1\") "
Sep 11 17:00:12.310459 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310318    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home" (OuterVolumeSpecName: "home") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:12.310459 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310373    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:12.310561 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310510    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:12.310561 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310532    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.310561 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.310550    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.312254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.312233    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:12.312480 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.312453    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm" (OuterVolumeSpecName: "dshm") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:12.312480 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.312468    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg" (OuterVolumeSpecName: "kube-api-access-chdsg") pod "c8bfaeef-0038-4524-b1be-48457f0695d1" (UID: "c8bfaeef-0038-4524-b1be-48457f0695d1"). InnerVolumeSpecName "kube-api-access-chdsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:12.411110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.411066    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8bfaeef-0038-4524-b1be-48457f0695d1-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.411110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.411107    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.411110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.411115    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8bfaeef-0038-4524-b1be-48457f0695d1-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.411110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.411124    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdsg\" (UniqueName: \"kubernetes.io/projected/c8bfaeef-0038-4524-b1be-48457f0695d1-kube-api-access-chdsg\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.734739 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.734702    2550 generic.go:356] "Generic (PLEG): container finished" podID="c8bfaeef-0038-4524-b1be-48457f0695d1" containerID="71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1" exitCode=0
Sep 11 17:00:12.734926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.734754    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"
Sep 11 17:00:12.734926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.734768    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" event={"ID":"c8bfaeef-0038-4524-b1be-48457f0695d1","Type":"ContainerDied","Data":"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"}
Sep 11 17:00:12.734926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.734819    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw" event={"ID":"c8bfaeef-0038-4524-b1be-48457f0695d1","Type":"ContainerDied","Data":"65f341102149e72d7af8f4e927adfe73f4a157cbac2c446591cbbdd9ce84cca9"}
Sep 11 17:00:12.734926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.734847    2550 scope.go:117] "RemoveContainer" containerID="71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"
Sep 11 17:00:12.743940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.743914    2550 scope.go:117] "RemoveContainer" containerID="71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"
Sep 11 17:00:12.744219 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:00:12.744196    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1\": container with ID starting with 71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1 not found: ID does not exist" containerID="71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"
Sep 11 17:00:12.744308 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.744227    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1"} err="failed to get container status \"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1\": rpc error: code = NotFound desc = could not find container \"71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1\": container with ID starting with 71d402662acf7a25953153ac47407f0c7584937bbe5e88c1d06ceceb59330ff1 not found: ID does not exist"
Sep 11 17:00:12.756197 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.756172    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 17:00:12.758696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:12.758671    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-q9qgw"]
Sep 11 17:00:13.687207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:13.687172    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8bfaeef-0038-4524-b1be-48457f0695d1" path="/var/lib/kubelet/pods/c8bfaeef-0038-4524-b1be-48457f0695d1/volumes"
Sep 11 17:00:19.388955 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.388925    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:00:19.389482 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.389325    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8bfaeef-0038-4524-b1be-48457f0695d1" containerName="main"
Sep 11 17:00:19.389482 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.389338    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8bfaeef-0038-4524-b1be-48457f0695d1" containerName="main"
Sep 11 17:00:19.389482 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.389405    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8bfaeef-0038-4524-b1be-48457f0695d1" containerName="main"
Sep 11 17:00:19.392131 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.392115    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.396310 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.396283    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 11 17:00:19.396310 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.396283    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 11 17:00:19.396509 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.396290    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-v8pph\""
Sep 11 17:00:19.396509 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.396338    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 11 17:00:19.401230 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.401204    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:00:19.408385 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.408353    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:19.473649 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473593    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.473849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473653    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcgh\" (UniqueName: \"kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.473849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473739    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.473849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473770    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.473982 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.473982 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.473950    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575263    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575303    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcgh\" (UniqueName: \"kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575565 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575333    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575565 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575357    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575565 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575407    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575565 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575464    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575800    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.575885 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575834    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.576011 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.575989    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.578379 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.578352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.578604 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.578587    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.584314 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.584288    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcgh\" (UniqueName: \"kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh\") pod \"delw-v1-kserve-6cf8cbb847-ls927\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.704219 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.704186    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:19.974207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:19.974122    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:00:19.977922 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:00:19.977886    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7fd259_db35_45de_9397_91782ac7241c.slice/crio-6d87ee29aa5bb67fa7cdecbf3260f741611dd2fe1298b20ac15222d1f002630c WatchSource:0}: Error finding container 6d87ee29aa5bb67fa7cdecbf3260f741611dd2fe1298b20ac15222d1f002630c: Status 404 returned error can't find the container with id 6d87ee29aa5bb67fa7cdecbf3260f741611dd2fe1298b20ac15222d1f002630c
Sep 11 17:00:20.775931 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:20.775895    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" event={"ID":"7f7fd259-db35-45de-9397-91782ac7241c","Type":"ContainerStarted","Data":"a5206df67a5fa952044a72559ad9090ab752ff35ec87402281eed803b95b86bb"}
Sep 11 17:00:20.776330 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:20.775946    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" event={"ID":"7f7fd259-db35-45de-9397-91782ac7241c","Type":"ContainerStarted","Data":"6d87ee29aa5bb67fa7cdecbf3260f741611dd2fe1298b20ac15222d1f002630c"}
Sep 11 17:00:20.794008 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:20.793950    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" podStartSLOduration=1.793934184 podStartE2EDuration="1.793934184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:20.791817395 +0000 UTC m=+2611.566517997" watchObservedRunningTime="2026-09-11 17:00:20.793934184 +0000 UTC m=+2611.568634762"
Sep 11 17:00:21.704558 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:21.704507    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:24.907602 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:24.907569    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:29.704332 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:29.704296    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:29.717286 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:29.717259    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:29.829456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:29.829425    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:00:33.808400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:33.808361    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:39.256303 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:39.256268    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:39.356429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:39.356381    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:50.371977 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:50.371891    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:58.395860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.395821    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:58.401950 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.401919    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 17:00:58.412144 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:00:58.412105    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:00:58.412144 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.412144 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.412144 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb" cmd=["/bin/sleep","15"]
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.412170    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" message="" err=<
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:00:58.412187    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main" containerID="cri-o://0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb"
Sep 11 17:00:58.412469 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.412231    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main" containerID="cri-o://0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb" gracePeriod=60
Sep 11 17:00:58.474893 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.474857    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:00:58.616613 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.616565    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
Sep 11 17:00:58.955835 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.955798    2550 generic.go:356] "Generic (PLEG): container finished" podID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerID="0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb" exitCode=0
Sep 11 17:00:58.956039 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:58.955859    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerDied","Data":"0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb"}
Sep 11 17:00:59.172662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.172611    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 17:00:59.217298 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217220    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217298 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217259    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217298 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217291    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217327    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217369    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217399    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl5fv\" (UniqueName: \"kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217447    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217471    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home\") pod \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\" (UID: \"f0147e42-ec2c-4f1f-addf-a678d3bfdb35\") "
Sep 11 17:00:59.217596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217535    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.217937 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217818    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.217937 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217839    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.218020 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.217996    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home" (OuterVolumeSpecName: "home") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.218060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.218039    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:59.219906 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.219884    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm" (OuterVolumeSpecName: "dshm") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.220014 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.219930    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:59.220199 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.220180    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv" (OuterVolumeSpecName: "kube-api-access-xl5fv") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "kube-api-access-xl5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:59.245737 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.245712    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0147e42-ec2c-4f1f-addf-a678d3bfdb35" (UID: "f0147e42-ec2c-4f1f-addf-a678d3bfdb35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.318302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318268    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318295    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5fv\" (UniqueName: \"kubernetes.io/projected/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kube-api-access-xl5fv\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318306    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318489 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318317    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318489 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318326    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318489 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318334    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.318489 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.318341    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0147e42-ec2c-4f1f-addf-a678d3bfdb35-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.962513 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.962442    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"
Sep 11 17:00:59.962940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.962433    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc" event={"ID":"f0147e42-ec2c-4f1f-addf-a678d3bfdb35","Type":"ContainerDied","Data":"b3f15e0d658ba2d267105a473e2e9f729c8ef2958a4e05bf6b66387f7a8bfe67"}
Sep 11 17:00:59.962940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.962577    2550 scope.go:117] "RemoveContainer" containerID="0b8b86c2a1d53f136b72043badcb74864e312bf58d3667682fb5f1762bce78fb"
Sep 11 17:00:59.972248 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.972229    2550 scope.go:117] "RemoveContainer" containerID="947e5a8c26872874a785b2e6fc02327afeb923c4c5ef9a7a6f7d0b3b18cf932b"
Sep 11 17:00:59.980897 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.980875    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 17:00:59.985136 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:00:59.985115    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-jj4gc"]
Sep 11 17:01:01.686601 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:01.686565    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" path="/var/lib/kubelet/pods/f0147e42-ec2c-4f1f-addf-a678d3bfdb35/volumes"
Sep 11 17:01:08.501260 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:08.501214    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:24.781735 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:24.781702    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:24.896782 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:24.896745    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:24.909321 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:24.909291    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:01:24.995236 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:24.995208    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:29.860428 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:29.860374    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:30.262532 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.262502    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"]
Sep 11 17:01:30.263007 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.262988    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main"
Sep 11 17:01:30.263118 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.263009    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main"
Sep 11 17:01:30.263118 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.263028    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="storage-initializer"
Sep 11 17:01:30.263118 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.263037    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="storage-initializer"
Sep 11 17:01:30.263278 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.263138    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0147e42-ec2c-4f1f-addf-a678d3bfdb35" containerName="main"
Sep 11 17:01:30.265786 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.265764    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.268716 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.268695    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 11 17:01:30.269884 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.269858    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 11 17:01:30.270002 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.269928    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 11 17:01:30.270002 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.269957    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-7tqx5\""
Sep 11 17:01:30.276998 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.276975    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"]
Sep 11 17:01:30.299117 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.299083    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:30.370660 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370611    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95lh\" (UniqueName: \"kubernetes.io/projected/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-kube-api-access-v95lh\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.370801 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370681    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.370801 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370705    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.370930 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370817    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.370930 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370880    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-home\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.371020 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.370930    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472149 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472118    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472163    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-home\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472288    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472434 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472350    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95lh\" (UniqueName: \"kubernetes.io/projected/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-kube-api-access-v95lh\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472434 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472386    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472434 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472402    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472599 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472542    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-home\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472686 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472668    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.472740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.472707    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.474548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.474525    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-dshm\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.474835 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.474819    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.480507 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.480487    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95lh\" (UniqueName: \"kubernetes.io/projected/0c4875a2-3b4f-4d78-8d4a-48d46ad6728e-kube-api-access-v95lh\") pod \"roll-v1-kserve-69cf8bcfc-xf5gt\" (UID: \"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.578826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.578762    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:30.849673 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:30.849645    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"]
Sep 11 17:01:30.851911 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:01:30.851884    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4875a2_3b4f_4d78_8d4a_48d46ad6728e.slice/crio-85c5adedfa190906dffcde364b48f6642fb6876cda17611c8f7bf83368d9d575 WatchSource:0}: Error finding container 85c5adedfa190906dffcde364b48f6642fb6876cda17611c8f7bf83368d9d575: Status 404 returned error can't find the container with id 85c5adedfa190906dffcde364b48f6642fb6876cda17611c8f7bf83368d9d575
Sep 11 17:01:31.106307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:31.106220    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt" event={"ID":"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e","Type":"ContainerStarted","Data":"4bab1dc609ce84649f1e6a11ca1f141454eeecb04cbaa034bd4b8bb2b7de1db2"}
Sep 11 17:01:31.106307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:31.106283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt" event={"ID":"0c4875a2-3b4f-4d78-8d4a-48d46ad6728e","Type":"ContainerStarted","Data":"85c5adedfa190906dffcde364b48f6642fb6876cda17611c8f7bf83368d9d575"}
Sep 11 17:01:31.126894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:31.126851    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt" podStartSLOduration=1.126836166 podStartE2EDuration="1.126836166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:31.124465665 +0000 UTC m=+2681.899166246" watchObservedRunningTime="2026-09-11 17:01:31.126836166 +0000 UTC m=+2681.901536742"
Sep 11 17:01:31.579138 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:31.579098    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:34.577128 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.577095    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"]
Sep 11 17:01:34.580555 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.580533    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.583450 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.583428    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 11 17:01:34.588731 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.588708    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"]
Sep 11 17:01:34.610362 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.610341    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:34.709133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709104    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.709293 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709161    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8384de4d-3208-4343-9071-81ef550645a4-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.709293 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709195    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-home\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.709293 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709237    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.709448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709300    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxrw\" (UniqueName: \"kubernetes.io/projected/8384de4d-3208-4343-9071-81ef550645a4-kube-api-access-grxrw\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.709448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.709329    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.810679 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810645    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.810859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810696    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8384de4d-3208-4343-9071-81ef550645a4-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.810859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-home\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.810859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.811047 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810900    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxrw\" (UniqueName: \"kubernetes.io/projected/8384de4d-3208-4343-9071-81ef550645a4-kube-api-access-grxrw\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.811047 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.810934    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.811226 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.811201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-home\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.811306 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.811243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.811306 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.811249    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.812748 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.812726    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8384de4d-3208-4343-9071-81ef550645a4-dshm\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.813115 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.813097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8384de4d-3208-4343-9071-81ef550645a4-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.819270 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.819249    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxrw\" (UniqueName: \"kubernetes.io/projected/8384de4d-3208-4343-9071-81ef550645a4-kube-api-access-grxrw\") pod \"roll-v2-kserve-d4dbf9d67-4m8qr\" (UID: \"8384de4d-3208-4343-9071-81ef550645a4\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:34.891809 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:34.891740    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:35.163464 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:35.163442    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"]
Sep 11 17:01:35.165499 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:01:35.165465    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8384de4d_3208_4343_9071_81ef550645a4.slice/crio-30d4545a56095e3869bed86ece4f0ec56b2c9aaa90c4b4f44cad6b2f9e8d7044 WatchSource:0}: Error finding container 30d4545a56095e3869bed86ece4f0ec56b2c9aaa90c4b4f44cad6b2f9e8d7044: Status 404 returned error can't find the container with id 30d4545a56095e3869bed86ece4f0ec56b2c9aaa90c4b4f44cad6b2f9e8d7044
Sep 11 17:01:36.129077 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:36.129041    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr" event={"ID":"8384de4d-3208-4343-9071-81ef550645a4","Type":"ContainerStarted","Data":"9f4426be8c309177e496f944ff9eaccbef6d83435127d9b5a7f91f30cc6d973c"}
Sep 11 17:01:36.129469 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:36.129100    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr" event={"ID":"8384de4d-3208-4343-9071-81ef550645a4","Type":"ContainerStarted","Data":"30d4545a56095e3869bed86ece4f0ec56b2c9aaa90c4b4f44cad6b2f9e8d7044"}
Sep 11 17:01:36.150233 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:36.150185    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr" podStartSLOduration=2.150171998 podStartE2EDuration="2.150171998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:36.148130627 +0000 UTC m=+2686.922831208" watchObservedRunningTime="2026-09-11 17:01:36.150171998 +0000 UTC m=+2686.924872575"
Sep 11 17:01:36.892413 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:36.892383    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:39.928306 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:39.928264    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" podUID="7f7fd259-db35-45de-9397-91782ac7241c" containerName="main" containerID="cri-o://a5206df67a5fa952044a72559ad9090ab752ff35ec87402281eed803b95b86bb" gracePeriod=45
Sep 11 17:01:40.149908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.149871    2550 generic.go:356] "Generic (PLEG): container finished" podID="7f7fd259-db35-45de-9397-91782ac7241c" containerID="a5206df67a5fa952044a72559ad9090ab752ff35ec87402281eed803b95b86bb" exitCode=0
Sep 11 17:01:40.150050 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.149917    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" event={"ID":"7f7fd259-db35-45de-9397-91782ac7241c","Type":"ContainerDied","Data":"a5206df67a5fa952044a72559ad9090ab752ff35ec87402281eed803b95b86bb"}
Sep 11 17:01:40.186254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.186192    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:01:40.257722 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257687    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.257894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257741    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.257894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257761    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.257894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257779    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxcgh\" (UniqueName: \"kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.257894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257814    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.257894 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257843    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm\") pod \"7f7fd259-db35-45de-9397-91782ac7241c\" (UID: \"7f7fd259-db35-45de-9397-91782ac7241c\") "
Sep 11 17:01:40.258159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.257964    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:40.258159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.258031    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:40.258159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.258077    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home" (OuterVolumeSpecName: "home") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:40.258159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.258139    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.258159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.258157    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.259997 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.259974    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm" (OuterVolumeSpecName: "dshm") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:40.260113 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.259996    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:01:40.260113 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.260047    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh" (OuterVolumeSpecName: "kube-api-access-vxcgh") pod "7f7fd259-db35-45de-9397-91782ac7241c" (UID: "7f7fd259-db35-45de-9397-91782ac7241c"). InnerVolumeSpecName "kube-api-access-vxcgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:01:40.358583 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.358555    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.358583 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.358580    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7fd259-db35-45de-9397-91782ac7241c-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.358760 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.358589    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7fd259-db35-45de-9397-91782ac7241c-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.358760 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.358599    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxcgh\" (UniqueName: \"kubernetes.io/projected/7f7fd259-db35-45de-9397-91782ac7241c-kube-api-access-vxcgh\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:01:40.579408 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.579370    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:40.591846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:40.591822    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:41.156253 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.156223    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"
Sep 11 17:01:41.156775 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.156265    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927" event={"ID":"7f7fd259-db35-45de-9397-91782ac7241c","Type":"ContainerDied","Data":"6d87ee29aa5bb67fa7cdecbf3260f741611dd2fe1298b20ac15222d1f002630c"}
Sep 11 17:01:41.156775 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.156330    2550 scope.go:117] "RemoveContainer" containerID="a5206df67a5fa952044a72559ad9090ab752ff35ec87402281eed803b95b86bb"
Sep 11 17:01:41.168317 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.168298    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-xf5gt"
Sep 11 17:01:41.179130 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.179107    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:01:41.185107 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.185083    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-ls927"]
Sep 11 17:01:41.686374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:41.686340    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7fd259-db35-45de-9397-91782ac7241c" path="/var/lib/kubelet/pods/7f7fd259-db35-45de-9397-91782ac7241c/volumes"
Sep 11 17:01:44.892718 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:44.892619    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:44.905037 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:44.905012    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:45.186432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:45.186347    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-4m8qr"
Sep 11 17:01:48.257017 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:48.256978    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:48.356562 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:48.356532    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:54.223012 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:54.222976    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:59.792460 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:59.792421    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:01:59.892290 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:01:59.892256    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:06.202681 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.202640    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:06.399049 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.399014    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:02:06.399400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.399385    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7fd259-db35-45de-9397-91782ac7241c" containerName="main"
Sep 11 17:02:06.399400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.399400    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7fd259-db35-45de-9397-91782ac7241c" containerName="main"
Sep 11 17:02:06.399558 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.399459    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7fd259-db35-45de-9397-91782ac7241c" containerName="main"
Sep 11 17:02:06.402507 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.402484    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.405440 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.405407    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-8dsjb\""
Sep 11 17:02:06.405568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.405485    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 11 17:02:06.405568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.405511    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 11 17:02:06.405568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.405485    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-z4m6z\""
Sep 11 17:02:06.405568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.405464    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 11 17:02:06.411811 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.411783    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:02:06.437521 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.437494    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:06.476751 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.476692    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.476751 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.476731    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkgf\" (UniqueName: \"kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.577885 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.577854    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.578018 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.577904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkgf\" (UniqueName: \"kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.580317 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.580294    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.586193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.586172    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkgf\" (UniqueName: \"kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.714127 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.714100    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:06.988860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:06.988834    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:02:06.990637 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:02:06.990592    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b6c7c2_5cbd_4c51_9792_09cc0452a71b.slice/crio-891d278328ed9ec8df178f790fa07bdffdacedcc4ff922f80829d3fde04f9e34 WatchSource:0}: Error finding container 891d278328ed9ec8df178f790fa07bdffdacedcc4ff922f80829d3fde04f9e34: Status 404 returned error can't find the container with id 891d278328ed9ec8df178f790fa07bdffdacedcc4ff922f80829d3fde04f9e34
Sep 11 17:02:07.282016 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:07.281917    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:07.284691 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:07.284667    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" event={"ID":"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b","Type":"ContainerStarted","Data":"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"}
Sep 11 17:02:07.284832 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:07.284719    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" event={"ID":"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b","Type":"ContainerStarted","Data":"891d278328ed9ec8df178f790fa07bdffdacedcc4ff922f80829d3fde04f9e34"}
Sep 11 17:02:07.284832 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:07.284798    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:07.301961 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:07.301921    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podStartSLOduration=1.301907615 podStartE2EDuration="1.301907615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:02:07.30066521 +0000 UTC m=+2718.075365786" watchObservedRunningTime="2026-09-11 17:02:07.301907615 +0000 UTC m=+2718.076608191"
Sep 11 17:02:12.818597 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:12.818520    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:12.879346 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:12.879310    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:13.683681 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:13.683646    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:38.291301 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:38.291272    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:02:39.808635 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:39.808589    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:39.819403 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:39.819371    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:02:39.886555 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:39.886514    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:48.291808 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:48.291768    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:02:50.714593 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:50.714551    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v1-kserve-69cf8bcfc-xf5gt_0c4875a2-3b4f-4d78-8d4a-48d46ad6728e/main/0.log"
Sep 11 17:02:51.796129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:51.796093    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-rollback-b8fdb441_roll-v2-kserve-d4dbf9d67-4m8qr_8384de4d-3208-4343-9071-81ef550645a4/main/0.log"
Sep 11 17:02:52.945790 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:52.945681    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:53.047604 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:53.047564    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:02:54.839960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:54.839919    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main" containerID="cri-o://8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc" gracePeriod=45
Sep 11 17:02:58.291873 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:02:58.291834    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:03:07.978230 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:07.978191    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:08.291931 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:08.291843    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:03:08.292082 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:08.291935    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:03:18.292252 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:18.292167    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:03:22.419400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:22.419363    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:25.082480 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.082456    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:03:25.255904 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.255867    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs\") pod \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") "
Sep 11 17:03:25.256110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.255910    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzkgf\" (UniqueName: \"kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf\") pod \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\" (UID: \"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b\") "
Sep 11 17:03:25.258203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.258162    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf" (OuterVolumeSpecName: "kube-api-access-kzkgf") pod "f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" (UID: "f8b6c7c2-5cbd-4c51-9792-09cc0452a71b"). InnerVolumeSpecName "kube-api-access-kzkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:03:25.258203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.258189    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" (UID: "f8b6c7c2-5cbd-4c51-9792-09cc0452a71b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:03:25.357056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.357029    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:03:25.357056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.357053    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkgf\" (UniqueName: \"kubernetes.io/projected/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b-kube-api-access-kzkgf\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:03:25.650221 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.650131    2550 generic.go:356] "Generic (PLEG): container finished" podID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerID="8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc" exitCode=0
Sep 11 17:03:25.650221 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.650185    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"
Sep 11 17:03:25.650430 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.650222    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" event={"ID":"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b","Type":"ContainerDied","Data":"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"}
Sep 11 17:03:25.650430 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.650310    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg" event={"ID":"f8b6c7c2-5cbd-4c51-9792-09cc0452a71b","Type":"ContainerDied","Data":"891d278328ed9ec8df178f790fa07bdffdacedcc4ff922f80829d3fde04f9e34"}
Sep 11 17:03:25.650430 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.650339    2550 scope.go:117] "RemoveContainer" containerID="8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"
Sep 11 17:03:25.660063 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.660044    2550 scope.go:117] "RemoveContainer" containerID="8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"
Sep 11 17:03:25.660300 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:03:25.660280    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc\": container with ID starting with 8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc not found: ID does not exist" containerID="8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"
Sep 11 17:03:25.660366 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.660307    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc"} err="failed to get container status \"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc\": rpc error: code = NotFound desc = could not find container \"8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc\": container with ID starting with 8930c89055b35a59d4d3fa0dc2d3810379fcb58db4010d5222021f7e46b8b9dc not found: ID does not exist"
Sep 11 17:03:25.671818 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.671788    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:03:25.677603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.677582    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheb9xpg"]
Sep 11 17:03:25.687291 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:25.687268    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" path="/var/lib/kubelet/pods/f8b6c7c2-5cbd-4c51-9792-09cc0452a71b/volumes"
Sep 11 17:03:29.815395 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:29.815362    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:29.912388 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:29.912357    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:35.764862 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:35.764829    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:40.066379 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:40.066343    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:41.573152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:41.573119    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:41.702680 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:41.702650    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:56.011031 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.010993    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:56.187647 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.187588    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:03:56.188001 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.187982    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main"
Sep 11 17:03:56.188056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.188004    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main"
Sep 11 17:03:56.188151 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.188139    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b6c7c2-5cbd-4c51-9792-09cc0452a71b" containerName="main"
Sep 11 17:03:56.192593 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.192570    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.196644 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.196595    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 11 17:03:56.196644 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.196607    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-ptrfr\""
Sep 11 17:03:56.196824 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.196596    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 11 17:03:56.196824 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.196656    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 11 17:03:56.196824 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.196781    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-2nw56\""
Sep 11 17:03:56.202779 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.202754    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:03:56.217710 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.217674    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:03:56.314690 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.314601    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh86\" (UniqueName: \"kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.314690 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.314673    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.415912 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.415878    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh86\" (UniqueName: \"kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.416085 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.415921    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.418429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.418402    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.423815 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.423790    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh86\" (UniqueName: \"kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.504328 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.504293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:56.774419 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.774220    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:03:56.776657 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:03:56.776602    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d4685b_71fa_4448_a63c_49b124e84c13.slice/crio-fb3dcf402635e358d727165ff4390a11d0e13e80b5e2859daa04e99d009bc92b WatchSource:0}: Error finding container fb3dcf402635e358d727165ff4390a11d0e13e80b5e2859daa04e99d009bc92b: Status 404 returned error can't find the container with id fb3dcf402635e358d727165ff4390a11d0e13e80b5e2859daa04e99d009bc92b
Sep 11 17:03:56.794143 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:56.794060    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" event={"ID":"c6d4685b-71fa-4448-a63c-49b124e84c13","Type":"ContainerStarted","Data":"fb3dcf402635e358d727165ff4390a11d0e13e80b5e2859daa04e99d009bc92b"}
Sep 11 17:03:57.801760 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:57.801714    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" event={"ID":"c6d4685b-71fa-4448-a63c-49b124e84c13","Type":"ContainerStarted","Data":"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"}
Sep 11 17:03:57.802221 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:57.801856    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:03:57.825143 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:03:57.825094    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podStartSLOduration=1.825081019 podStartE2EDuration="1.825081019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:57.822937172 +0000 UTC m=+2828.597637751" watchObservedRunningTime="2026-09-11 17:03:57.825081019 +0000 UTC m=+2828.599781597"
Sep 11 17:04:20.367409 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:20.367373    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:23.767556 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:23.767523    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:28.808412 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:28.808377    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:04:45.384609 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:45.384524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:45.480336 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:45.480300    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:45.646138 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:45.646058    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:45.742461 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:45.742430    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:46.380754 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:46.380717    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:48.872561 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:48.872525    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:50.409213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:50.409175    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:57.669774 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:57.669743    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:04:57.985767 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:04:57.985726    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:04.582198 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:04.582158    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:24.376946 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.376907    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:24.381902 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.381876    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.388060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.387835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 11 17:05:24.388060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.387884    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 11 17:05:24.388060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.387916    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-wbk86\""
Sep 11 17:05:24.388060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.387835    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 11 17:05:24.390599 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.390574    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:24.418035 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.417989    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:24.468342 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.468486 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468349    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.468486 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.468486 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.468595 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468511    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.468595 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.468584    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpss\" (UniqueName: \"kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.569952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.569918    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpss\" (UniqueName: \"kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.569973    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.569999    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570027    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570458 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570432    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570530 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570460    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.570592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.570551    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.572450 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.572429    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.572450 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.572442    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.577563 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.577536    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpss\" (UniqueName: \"kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.697463 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.697440    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:24.967309 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:24.967284    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:24.970254 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:05:24.970215    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8f4316_dae3_43a3_aab0_5faf3350ec79.slice/crio-ecd8547ecbc88fe199e620aa9f0a9f0392a72c2ed154708dc88bbc8b7b859108 WatchSource:0}: Error finding container ecd8547ecbc88fe199e620aa9f0a9f0392a72c2ed154708dc88bbc8b7b859108: Status 404 returned error can't find the container with id ecd8547ecbc88fe199e620aa9f0a9f0392a72c2ed154708dc88bbc8b7b859108
Sep 11 17:05:25.218292 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:25.218200    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" event={"ID":"ee8f4316-dae3-43a3-aab0-5faf3350ec79","Type":"ContainerStarted","Data":"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"}
Sep 11 17:05:25.218292 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:25.218259    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" event={"ID":"ee8f4316-dae3-43a3-aab0-5faf3350ec79","Type":"ContainerStarted","Data":"ecd8547ecbc88fe199e620aa9f0a9f0392a72c2ed154708dc88bbc8b7b859108"}
Sep 11 17:05:25.236772 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:25.236726    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" podStartSLOduration=1.236710918 podStartE2EDuration="1.236710918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:05:25.235159572 +0000 UTC m=+2916.009860152" watchObservedRunningTime="2026-09-11 17:05:25.236710918 +0000 UTC m=+2916.011411496"
Sep 11 17:05:25.698488 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:25.698458    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:27.528692 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:27.528647    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:33.009984 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:33.009941    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:33.066603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:33.066571    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:34.698173 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:34.698139    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:34.711482 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:34.711457    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:35.280729 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:35.280702    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:35.729269 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:35.729238    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:50.909017 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:50.908982    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:51.007529 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:51.007481    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:54.393315 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.393283    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:54.402734 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:05:54.402698    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:05:54.402734 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:54.402734 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:54.402734 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef" cmd=["/bin/sleep","15"]
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.402757    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" message="" err=<
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:05:54.402774    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerName="main" containerID="cri-o://15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"
Sep 11 17:05:54.402987 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.402815    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerName="main" containerID="cri-o://15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef" gracePeriod=60
Sep 11 17:05:54.414411 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.414383    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:54.745437 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.745407    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:54.831859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.831829    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832017 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.831902    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832017 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.831925    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnpss\" (UniqueName: \"kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832017 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.831965    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832172 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832017    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832172 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832045    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs\") pod \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\" (UID: \"ee8f4316-dae3-43a3-aab0-5faf3350ec79\") "
Sep 11 17:05:54.832172 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832064    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:54.832334 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832211    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home" (OuterVolumeSpecName: "home") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:54.832334 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832246    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:54.832448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832389    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:54.832448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832407    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:54.832448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.832420    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:54.834252 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.834226    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm" (OuterVolumeSpecName: "dshm") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:54.834363 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.834263    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:54.834416 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.834367    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss" (OuterVolumeSpecName: "kube-api-access-dnpss") pod "ee8f4316-dae3-43a3-aab0-5faf3350ec79" (UID: "ee8f4316-dae3-43a3-aab0-5faf3350ec79"). InnerVolumeSpecName "kube-api-access-dnpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:54.933329 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.933268    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8f4316-dae3-43a3-aab0-5faf3350ec79-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:54.933329 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.933292    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8f4316-dae3-43a3-aab0-5faf3350ec79-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:54.933329 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:54.933302    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnpss\" (UniqueName: \"kubernetes.io/projected/ee8f4316-dae3-43a3-aab0-5faf3350ec79-kube-api-access-dnpss\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:05:55.368589 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.368554    2550 generic.go:356] "Generic (PLEG): container finished" podID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerID="15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef" exitCode=0
Sep 11 17:05:55.368784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.368603    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"
Sep 11 17:05:55.368784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.368610    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" event={"ID":"ee8f4316-dae3-43a3-aab0-5faf3350ec79","Type":"ContainerDied","Data":"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"}
Sep 11 17:05:55.368784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.368671    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh" event={"ID":"ee8f4316-dae3-43a3-aab0-5faf3350ec79","Type":"ContainerDied","Data":"ecd8547ecbc88fe199e620aa9f0a9f0392a72c2ed154708dc88bbc8b7b859108"}
Sep 11 17:05:55.368784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.368693    2550 scope.go:117] "RemoveContainer" containerID="15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"
Sep 11 17:05:55.378293 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.378274    2550 scope.go:117] "RemoveContainer" containerID="15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"
Sep 11 17:05:55.378580 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:05:55.378561    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef\": container with ID starting with 15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef not found: ID does not exist" containerID="15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"
Sep 11 17:05:55.378654 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.378587    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef"} err="failed to get container status \"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef\": rpc error: code = NotFound desc = could not find container \"15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef\": container with ID starting with 15097d021c5c958b0342496cbdfa2cca9e42b927f7fa3e77a207f6706c1bb6ef not found: ID does not exist"
Sep 11 17:05:55.390603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.390581    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:55.394395 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.394372    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qxffh"]
Sep 11 17:05:55.416204 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.416184    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:55.687487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:55.687413    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" path="/var/lib/kubelet/pods/ee8f4316-dae3-43a3-aab0-5faf3350ec79/volumes"
Sep 11 17:05:57.185009 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:57.184979    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:05:57.281764 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:05:57.281730    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:02.177176 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:02.177144    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:07.568471 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:07.568431    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:07.970826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:07.970777    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:11.704231 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:11.704197    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:13.805010 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:13.804923    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:23.502460 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:23.502425    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:24.158926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:24.158873    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:29.876165 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:29.876124    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:38.654130 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:38.654098    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:06:38.656537 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:38.656496    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:38.715745 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:38.715718    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:38.808101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:38.808065    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:48.808556 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:48.808517    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:49.858120 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:49.858088    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:52.669589 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:52.669553    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:52.738866 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:52.738838    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:06:53.680636 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:53.680582    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" containerID="cri-o://700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c" gracePeriod=45
Sep 11 17:06:58.808265 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:58.808226    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:58.808674 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:06:58.808350    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:07:08.181897 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:08.181856    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:08.808396 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:08.808355    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:07:18.807955 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:18.807918    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:07:23.926291 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:23.926259    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:07:23.965520 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:23.965495    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs\") pod \"c6d4685b-71fa-4448-a63c-49b124e84c13\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") "
Sep 11 17:07:23.965520 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:23.965528    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlh86\" (UniqueName: \"kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86\") pod \"c6d4685b-71fa-4448-a63c-49b124e84c13\" (UID: \"c6d4685b-71fa-4448-a63c-49b124e84c13\") "
Sep 11 17:07:23.967679 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:23.967648    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86" (OuterVolumeSpecName: "kube-api-access-hlh86") pod "c6d4685b-71fa-4448-a63c-49b124e84c13" (UID: "c6d4685b-71fa-4448-a63c-49b124e84c13"). InnerVolumeSpecName "kube-api-access-hlh86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:07:23.967796 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:23.967663    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6d4685b-71fa-4448-a63c-49b124e84c13" (UID: "c6d4685b-71fa-4448-a63c-49b124e84c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:07:24.066794 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.066762    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d4685b-71fa-4448-a63c-49b124e84c13-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:07:24.066794 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.066789    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlh86\" (UniqueName: \"kubernetes.io/projected/c6d4685b-71fa-4448-a63c-49b124e84c13-kube-api-access-hlh86\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:07:24.781935 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.781897    2550 generic.go:356] "Generic (PLEG): container finished" podID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerID="700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c" exitCode=0
Sep 11 17:07:24.781935 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.781937    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"
Sep 11 17:07:24.782173 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.781978    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" event={"ID":"c6d4685b-71fa-4448-a63c-49b124e84c13","Type":"ContainerDied","Data":"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"}
Sep 11 17:07:24.782173 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.782025    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst" event={"ID":"c6d4685b-71fa-4448-a63c-49b124e84c13","Type":"ContainerDied","Data":"fb3dcf402635e358d727165ff4390a11d0e13e80b5e2859daa04e99d009bc92b"}
Sep 11 17:07:24.782173 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.782047    2550 scope.go:117] "RemoveContainer" containerID="700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"
Sep 11 17:07:24.791528 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.791388    2550 scope.go:117] "RemoveContainer" containerID="700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"
Sep 11 17:07:24.791678 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:07:24.791657    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c\": container with ID starting with 700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c not found: ID does not exist" containerID="700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"
Sep 11 17:07:24.791730 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.791687    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c"} err="failed to get container status \"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c\": rpc error: code = NotFound desc = could not find container \"700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c\": container with ID starting with 700627187f935369ff7ef94ba0c8a1f48914a99260f70e61a751affbb26a623c not found: ID does not exist"
Sep 11 17:07:24.804213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.804191    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:07:24.808953 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:24.808932    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche8hgst"]
Sep 11 17:07:25.686755 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:25.686723    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" path="/var/lib/kubelet/pods/c6d4685b-71fa-4448-a63c-49b124e84c13/volumes"
Sep 11 17:07:28.740240 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:28.740206    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:28.834400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:28.834362    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:34.816993 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:34.816956    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:40.239830 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:40.239778    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:40.304864 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:40.304831    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:43.699606 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.699515    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:07:43.700096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700045    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main"
Sep 11 17:07:43.700096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700063    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main"
Sep 11 17:07:43.700096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700073    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerName="main"
Sep 11 17:07:43.700096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700081    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerName="main"
Sep 11 17:07:43.700329 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700190    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d4685b-71fa-4448-a63c-49b124e84c13" containerName="main"
Sep 11 17:07:43.700329 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.700209    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8f4316-dae3-43a3-aab0-5faf3350ec79" containerName="main"
Sep 11 17:07:43.702973 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.702953    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.705670 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.705650    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 11 17:07:43.705870 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.705839    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-fz682\""
Sep 11 17:07:43.705870 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.705848    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 11 17:07:43.706050 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.705893    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 11 17:07:43.710486 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.710431    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:07:43.781735 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.781702    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:43.828785 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828759    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.828896 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.828896 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92fs\" (UniqueName: \"kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.828969 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828896    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.828969 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.829039 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.828976    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930057 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930020    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930065    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930090    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930160    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930254 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930187    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92fs\" (UniqueName: \"kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930530 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930500    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930594 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930538    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.930750 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.930728    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.932436 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.932408    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.933072 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.933048    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.939407 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.939377    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92fs\" (UniqueName: \"kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:43.968879 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:43.968794    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:44.014686 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:44.014655    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:44.287728 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:44.287701    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:07:44.290096 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:07:44.290068    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode012d3f2_3e69_4663_bf78_e2e9136261bf.slice/crio-7d12a4dfba801d6ed3cc1da8a69a2fb20714d8b0b6ef3a5c1073b26689b0fe5b WatchSource:0}: Error finding container 7d12a4dfba801d6ed3cc1da8a69a2fb20714d8b0b6ef3a5c1073b26689b0fe5b: Status 404 returned error can't find the container with id 7d12a4dfba801d6ed3cc1da8a69a2fb20714d8b0b6ef3a5c1073b26689b0fe5b
Sep 11 17:07:44.882283 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:44.882243    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" event={"ID":"e012d3f2-3e69-4663-bf78-e2e9136261bf","Type":"ContainerStarted","Data":"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"}
Sep 11 17:07:44.882696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:44.882303    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" event={"ID":"e012d3f2-3e69-4663-bf78-e2e9136261bf","Type":"ContainerStarted","Data":"7d12a4dfba801d6ed3cc1da8a69a2fb20714d8b0b6ef3a5c1073b26689b0fe5b"}
Sep 11 17:07:45.015026 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:45.015003    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:53.489803 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:53.489765    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:07:54.015771 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:54.015741    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:54.028012 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:54.027987    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:07:54.048130 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:54.048080    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" podStartSLOduration=11.048067442 podStartE2EDuration="11.048067442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:44.899736216 +0000 UTC m=+3055.674436796" watchObservedRunningTime="2026-09-11 17:07:54.048067442 +0000 UTC m=+3064.822768020"
Sep 11 17:07:54.937590 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:07:54.937558    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:08:05.177640 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:05.177580    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:12.618289 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:12.618252    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:12.682463 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:12.682437    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:24.415736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.415699    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:24.575761 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.575725    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:08:24.579414 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.579393    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.583479 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.583448    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-cdchp\""
Sep 11 17:08:24.583479 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.583461    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 11 17:08:24.589196 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.589169    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:08:24.608540 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.608511    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:24.665938 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.665867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qhp\" (UniqueName: \"kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.666055 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.665933    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.767265 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.767234    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.767436 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.767327    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qhp\" (UniqueName: \"kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.769935 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.769910    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.776246 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.776216    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qhp\" (UniqueName: \"kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:24.891300 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:24.891268    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:25.163696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:25.163664    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:08:25.164708 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:08:25.164684    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027bfa65_bdff_46e5_9b5f_84807a0841b0.slice/crio-aef326c271690922c465402672871f3156b2f89c4f8bf677002406c439393466 WatchSource:0}: Error finding container aef326c271690922c465402672871f3156b2f89c4f8bf677002406c439393466: Status 404 returned error can't find the container with id aef326c271690922c465402672871f3156b2f89c4f8bf677002406c439393466
Sep 11 17:08:26.075136 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:26.075101    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" event={"ID":"027bfa65-bdff-46e5-9b5f-84807a0841b0","Type":"ContainerStarted","Data":"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"}
Sep 11 17:08:26.075554 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:26.075157    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" event={"ID":"027bfa65-bdff-46e5-9b5f-84807a0841b0","Type":"ContainerStarted","Data":"aef326c271690922c465402672871f3156b2f89c4f8bf677002406c439393466"}
Sep 11 17:08:26.075554 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:26.075213    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:26.098392 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:26.098343    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podStartSLOduration=2.09832797 podStartE2EDuration="2.09832797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:08:26.096457694 +0000 UTC m=+3096.871158271" watchObservedRunningTime="2026-09-11 17:08:26.09832797 +0000 UTC m=+3096.873028547"
Sep 11 17:08:39.268798 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:39.268759    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:41.966514 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:41.966474    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:42.021152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:42.021122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:57.082021 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:57.081988    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:08:58.543579 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:58.543546    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:08:58.569786 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:58.569755    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:08:58.632125 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:08:58.632084    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:02.750584 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:02.750550    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:02.850542 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:02.850508    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:04.907807 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:04.907764    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:05.217006 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:05.216974    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:07.081683 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:07.081646    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:09:08.734888 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:08.734853    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:13.564768 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:13.564679    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main" containerID="cri-o://56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e" gracePeriod=45
Sep 11 17:09:14.496293 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:14.496256    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:14.557053 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:14.557023    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:17.081435 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:17.081380    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:09:19.182283 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:19.182250    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:27.081917 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:27.081875    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:09:27.082429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:27.082004    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:09:37.081959 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:37.081923    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:09:43.816572 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:43.816547    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:09:43.956054 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:43.956025    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs\") pod \"027bfa65-bdff-46e5-9b5f-84807a0841b0\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") "
Sep 11 17:09:43.956241 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:43.956114    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6qhp\" (UniqueName: \"kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp\") pod \"027bfa65-bdff-46e5-9b5f-84807a0841b0\" (UID: \"027bfa65-bdff-46e5-9b5f-84807a0841b0\") "
Sep 11 17:09:43.958200 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:43.958175    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "027bfa65-bdff-46e5-9b5f-84807a0841b0" (UID: "027bfa65-bdff-46e5-9b5f-84807a0841b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:09:43.958322 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:43.958302    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp" (OuterVolumeSpecName: "kube-api-access-p6qhp") pod "027bfa65-bdff-46e5-9b5f-84807a0841b0" (UID: "027bfa65-bdff-46e5-9b5f-84807a0841b0"). InnerVolumeSpecName "kube-api-access-p6qhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:09:44.057706 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.057673    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/027bfa65-bdff-46e5-9b5f-84807a0841b0-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:44.057706 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.057700    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qhp\" (UniqueName: \"kubernetes.io/projected/027bfa65-bdff-46e5-9b5f-84807a0841b0-kube-api-access-p6qhp\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:44.440668 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.440564    2550 generic.go:356] "Generic (PLEG): container finished" podID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerID="56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e" exitCode=0
Sep 11 17:09:44.440668 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.440606    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"
Sep 11 17:09:44.440668 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.440613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" event={"ID":"027bfa65-bdff-46e5-9b5f-84807a0841b0","Type":"ContainerDied","Data":"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"}
Sep 11 17:09:44.440668 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.440669    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v" event={"ID":"027bfa65-bdff-46e5-9b5f-84807a0841b0","Type":"ContainerDied","Data":"aef326c271690922c465402672871f3156b2f89c4f8bf677002406c439393466"}
Sep 11 17:09:44.440974 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.440691    2550 scope.go:117] "RemoveContainer" containerID="56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"
Sep 11 17:09:44.451611 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.451568    2550 scope.go:117] "RemoveContainer" containerID="56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"
Sep 11 17:09:44.452050 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:09:44.452030    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e\": container with ID starting with 56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e not found: ID does not exist" containerID="56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"
Sep 11 17:09:44.452129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.452060    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e"} err="failed to get container status \"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e\": rpc error: code = NotFound desc = could not find container \"56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e\": container with ID starting with 56c7c454ddad4ff6e639c6e7536e5dd960f801953a66d98514d3ef15614fa30e not found: ID does not exist"
Sep 11 17:09:44.462883 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.462859    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:09:44.468675 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:44.468653    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetg44v"]
Sep 11 17:09:45.686900 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:45.686866    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" path="/var/lib/kubelet/pods/027bfa65-bdff-46e5-9b5f-84807a0841b0/volumes"
Sep 11 17:09:48.727105 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.727071    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:09:48.736244 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.736215    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:48.737715 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:09:48.737679    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:09:48.737715 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:09:48.737715 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:09:48.737715 ip-10-0-135-175 kubenswrapper[2550]:  > containerID="99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7" cmd=["/bin/sleep","15"]
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.737739    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" message="" err=<
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:  >
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:09:48.737756    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main" containerID="cri-o://99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"
Sep 11 17:09:48.737918 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.737810    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main" containerID="cri-o://99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7" gracePeriod=60
Sep 11 17:09:48.797896 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.797866    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:09:48.927101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:48.927065    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 11 17:09:49.086697 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.086670    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:09:49.094546 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094519    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094670 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094562    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094670 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094593    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b92fs\" (UniqueName: \"kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094670 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094666    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094852 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094688    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094852 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094757    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm\") pod \"e012d3f2-3e69-4663-bf78-e2e9136261bf\" (UID: \"e012d3f2-3e69-4663-bf78-e2e9136261bf\") "
Sep 11 17:09:49.094852 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094785    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:09:49.095012 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094973    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:09:49.095012 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.094988    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home" (OuterVolumeSpecName: "home") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:09:49.095143 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.095121    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.095211 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.095145    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.095211 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.095155    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.096752 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.096731    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:09:49.096752 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.096743    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs" (OuterVolumeSpecName: "kube-api-access-b92fs") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "kube-api-access-b92fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:09:49.096873 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.096822    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm" (OuterVolumeSpecName: "dshm") pod "e012d3f2-3e69-4663-bf78-e2e9136261bf" (UID: "e012d3f2-3e69-4663-bf78-e2e9136261bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:09:49.195865 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.195826    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012d3f2-3e69-4663-bf78-e2e9136261bf-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.195865 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.195863    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012d3f2-3e69-4663-bf78-e2e9136261bf-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.196060 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.195878    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b92fs\" (UniqueName: \"kubernetes.io/projected/e012d3f2-3e69-4663-bf78-e2e9136261bf-kube-api-access-b92fs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:09:49.466657 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.466607    2550 generic.go:356] "Generic (PLEG): container finished" podID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerID="99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7" exitCode=0
Sep 11 17:09:49.466846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.466665    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"
Sep 11 17:09:49.466846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.466699    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" event={"ID":"e012d3f2-3e69-4663-bf78-e2e9136261bf","Type":"ContainerDied","Data":"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"}
Sep 11 17:09:49.466846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.466760    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl" event={"ID":"e012d3f2-3e69-4663-bf78-e2e9136261bf","Type":"ContainerDied","Data":"7d12a4dfba801d6ed3cc1da8a69a2fb20714d8b0b6ef3a5c1073b26689b0fe5b"}
Sep 11 17:09:49.466846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.466794    2550 scope.go:117] "RemoveContainer" containerID="99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"
Sep 11 17:09:49.476332 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.476313    2550 scope.go:117] "RemoveContainer" containerID="99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"
Sep 11 17:09:49.476603 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:09:49.476585    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7\": container with ID starting with 99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7 not found: ID does not exist" containerID="99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"
Sep 11 17:09:49.476695 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.476609    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7"} err="failed to get container status \"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7\": rpc error: code = NotFound desc = could not find container \"99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7\": container with ID starting with 99141911b3e1e29c96985f28dcd8ef0211b031547a3961cc3cdcf814e6b964e7 not found: ID does not exist"
Sep 11 17:09:49.488657 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.488617    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:09:49.492973 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.492951    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcpjl"]
Sep 11 17:09:49.688395 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:09:49.688354    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" path="/var/lib/kubelet/pods/e012d3f2-3e69-4663-bf78-e2e9136261bf/volumes"
Sep 11 17:10:38.842616 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:38.842584    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:40.139369 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:40.139334    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:45.183056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:45.182974    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:50.695271 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:50.695230    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:50.796010 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:50.795980    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:53.077856 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.077822    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078183    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main"
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078195    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main"
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078212    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main"
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078218    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main"
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078283    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e012d3f2-3e69-4663-bf78-e2e9136261bf" containerName="main"
Sep 11 17:10:53.078350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.078296    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="027bfa65-bdff-46e5-9b5f-84807a0841b0" containerName="main"
Sep 11 17:10:53.081379 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.081358    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.084271 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.084232    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 11 17:10:53.084271 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.084241    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 11 17:10:53.084492 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.084297    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-l44vr\""
Sep 11 17:10:53.084492 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.084303    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:10:53.084663 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.084647    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 11 17:10:53.085212 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.085194    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 11 17:10:53.096096 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.096073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:10:53.113855 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.113833    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.113984 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.113903    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.113984 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.113942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.113984 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.113966    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.114158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.114006    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.114158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.114032    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.114158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.114056    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hr66\" (UniqueName: \"kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.114158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.114083    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.114392 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.114371    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:53.214759 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214729    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.214905 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214762    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.214905 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214790    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.214905 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214814    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.214905 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hr66\" (UniqueName: \"kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.214905 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215192 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.214961    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215192 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215034    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215192 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215165    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215364 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215285    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215364 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215320    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215467 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215380    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.215611 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.215591    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.217141 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.217123    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.217559 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.217539    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.228213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.228190    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hr66\" (UniqueName: \"kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.317595 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.317564    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:10:53.392574 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.392505    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:10:53.668139 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.668116    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:10:53.670914 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:10:53.670886    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c940e0_a1ed_4560_8475_dc495077e179.slice/crio-52255ffbc4843188f30089de64ab27501cca4fa8a6e0f0cf142df6084e618fb7 WatchSource:0}: Error finding container 52255ffbc4843188f30089de64ab27501cca4fa8a6e0f0cf142df6084e618fb7: Status 404 returned error can't find the container with id 52255ffbc4843188f30089de64ab27501cca4fa8a6e0f0cf142df6084e618fb7
Sep 11 17:10:53.771168 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.771138    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerStarted","Data":"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"}
Sep 11 17:10:53.771304 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:53.771193    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerStarted","Data":"52255ffbc4843188f30089de64ab27501cca4fa8a6e0f0cf142df6084e618fb7"}
Sep 11 17:10:56.788840 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:56.788796    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1c940e0-a1ed-4560-8475-dc495077e179" containerID="5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676" exitCode=0
Sep 11 17:10:56.789284 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:56.788866    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerDied","Data":"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"}
Sep 11 17:10:57.795015 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:57.794975    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerStarted","Data":"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"}
Sep 11 17:10:57.820158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:57.820098    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podStartSLOduration=1.885230961 podStartE2EDuration="4.8200757s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.934844739s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:10:57.817481561 +0000 UTC m=+3248.592182138" watchObservedRunningTime="2026-09-11 17:10:57.8200757 +0000 UTC m=+3248.594776279"
Sep 11 17:10:58.392962 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:10:58.392933    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:11:03.393541 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:03.393502    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:11:03.394808 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:03.394778    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:11:07.382415 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:07.382367    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:11:13.393396 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:13.393350    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:11:13.814344 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:13.814305    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:11:13.912149 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:13.912114    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:11:23.393617 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:23.393573    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:11:33.393466 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:33.393419    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:11:43.393499 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:43.393458    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:11:53.393966 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:11:53.393925    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:12:03.393585 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:03.393540    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 11 17:12:03.739765 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:03.739724    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:03.833343 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:03.833305    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:09.586224 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:09.586182    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:13.403110 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:13.403035    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:12:13.411370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:13.411345    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:12:15.068725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:15.068688    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:15.171188 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:15.171150    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:16.524525 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:16.524493    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:16.785464 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:16.785359    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:23.385964 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:23.385919    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:44.325861 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:44.325825    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:12:45.435598 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:45.435546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:45.533255 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:45.533221    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:50.946741 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:50.946709    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:12:59.347975 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:59.347914    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main" containerID="cri-o://94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2" gracePeriod=45
Sep 11 17:12:59.993309 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:12:59.993285    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:13:00.073313 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073239    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073313 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073292    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073521 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073318    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073521 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073448    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073521 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073480    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073521 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073497    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hr66\" (UniqueName: \"kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073769 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073574    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073769 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073598    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir\") pod \"b1c940e0-a1ed-4560-8475-dc495077e179\" (UID: \"b1c940e0-a1ed-4560-8475-dc495077e179\") "
Sep 11 17:13:00.073920 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073889    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:00.073920 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.073899    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home" (OuterVolumeSpecName: "home") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:00.074079 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.074060    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:13:00.075688 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.075660    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:13:00.075826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.075729    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm" (OuterVolumeSpecName: "dshm") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:00.075915 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.075897    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66" (OuterVolumeSpecName: "kube-api-access-8hr66") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "kube-api-access-8hr66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:13:00.085436 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.085404    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:00.095386 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.095361    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1c940e0-a1ed-4560-8475-dc495077e179" (UID: "b1c940e0-a1ed-4560-8475-dc495077e179"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:00.174488 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174465    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174488 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174487    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1c940e0-a1ed-4560-8475-dc495077e179-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174497    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c940e0-a1ed-4560-8475-dc495077e179-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174504    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174512    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174521    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hr66\" (UniqueName: \"kubernetes.io/projected/b1c940e0-a1ed-4560-8475-dc495077e179-kube-api-access-8hr66\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174530    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.174651 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.174537    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c940e0-a1ed-4560-8475-dc495077e179-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:13:00.398177 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.398096    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1c940e0-a1ed-4560-8475-dc495077e179" containerID="94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2" exitCode=0
Sep 11 17:13:00.398177 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.398142    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"
Sep 11 17:13:00.398633 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.398176    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerDied","Data":"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"}
Sep 11 17:13:00.398633 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.398233    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz" event={"ID":"b1c940e0-a1ed-4560-8475-dc495077e179","Type":"ContainerDied","Data":"52255ffbc4843188f30089de64ab27501cca4fa8a6e0f0cf142df6084e618fb7"}
Sep 11 17:13:00.398633 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.398257    2550 scope.go:117] "RemoveContainer" containerID="94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"
Sep 11 17:13:00.409406 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.409380    2550 scope.go:117] "RemoveContainer" containerID="5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"
Sep 11 17:13:00.417784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.417763    2550 scope.go:117] "RemoveContainer" containerID="94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"
Sep 11 17:13:00.418039 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:13:00.418019    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2\": container with ID starting with 94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2 not found: ID does not exist" containerID="94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"
Sep 11 17:13:00.418103 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.418048    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2"} err="failed to get container status \"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2\": rpc error: code = NotFound desc = could not find container \"94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2\": container with ID starting with 94b1b824d2862b996159386f7b9a263c60af434c185a2fda67a014a4be1449f2 not found: ID does not exist"
Sep 11 17:13:00.418103 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.418065    2550 scope.go:117] "RemoveContainer" containerID="5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"
Sep 11 17:13:00.418325 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:13:00.418305    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676\": container with ID starting with 5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676 not found: ID does not exist" containerID="5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"
Sep 11 17:13:00.418366 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.418331    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676"} err="failed to get container status \"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676\": rpc error: code = NotFound desc = could not find container \"5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676\": container with ID starting with 5ad8aaa2d8a0e91e3c8ba71c5ca66e0ad96f88ab526a8f246fcb10af0a265676 not found: ID does not exist"
Sep 11 17:13:00.422457 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.422435    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:13:00.428341 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:00.428310    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b44874948-n5ccz"]
Sep 11 17:13:01.687207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:01.687174    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" path="/var/lib/kubelet/pods/b1c940e0-a1ed-4560-8475-dc495077e179/volumes"
Sep 11 17:13:05.682856 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:05.682815    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:12.321462 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:12.321427    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:17.491082 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:17.491039    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:34.304878 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:34.304835    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:34.396580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:34.396546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:40.353426 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:40.353377    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:42.261033 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:42.260913    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:42.318078 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:42.318047    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:45.767561 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:45.767527    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:45.867983 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:45.867946    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:48.798053 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798019    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:13:48.798487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798368    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="storage-initializer"
Sep 11 17:13:48.798487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798380    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="storage-initializer"
Sep 11 17:13:48.798487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798392    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main"
Sep 11 17:13:48.798487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798398    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main"
Sep 11 17:13:48.798487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.798464    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c940e0-a1ed-4560-8475-dc495077e179" containerName="main"
Sep 11 17:13:48.801437 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.801411    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.804438 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804413    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:13:48.804580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804438    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 11 17:13:48.804580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804417    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-lv2cl\""
Sep 11 17:13:48.804580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804440    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 11 17:13:48.804580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804413    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 11 17:13:48.804580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.804550    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 11 17:13:48.814276 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.814244    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:13:48.827418 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.827382    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:48.880191 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880160    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880324 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880212    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880324 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880289    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880351    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880406    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880525 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880441    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880525 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880486    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.880525 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.880517    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6sp\" (UniqueName: \"kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.980926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.980896    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.980940    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.980981    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981013    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981044    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981275 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981074    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6sp\" (UniqueName: \"kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981275 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981213    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981394 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981278    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981394 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981321    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981490 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981453    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981550 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981501    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981722 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981703    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.981835 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.981815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.983349 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.983329    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.983705 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.983689    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:48.991843 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:48.991822    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6sp\" (UniqueName: \"kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:49.070121 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:49.070025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:13:49.112589 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:49.112565    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:49.391736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:49.391481    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:13:49.394556 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:13:49.394527    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30719dfc_e23f_41e8_bf0c_20466335b7f1.slice/crio-334f595236e83da19915063f78c2b2a4dd81678747447dfa1cf190505ed597fb WatchSource:0}: Error finding container 334f595236e83da19915063f78c2b2a4dd81678747447dfa1cf190505ed597fb: Status 404 returned error can't find the container with id 334f595236e83da19915063f78c2b2a4dd81678747447dfa1cf190505ed597fb
Sep 11 17:13:49.632538 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:49.632506    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerStarted","Data":"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"}
Sep 11 17:13:49.632723 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:49.632556    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerStarted","Data":"334f595236e83da19915063f78c2b2a4dd81678747447dfa1cf190505ed597fb"}
Sep 11 17:13:52.647058 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:52.646975    2550 generic.go:356] "Generic (PLEG): container finished" podID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerID="655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c" exitCode=0
Sep 11 17:13:52.647058 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:52.647041    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerDied","Data":"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"}
Sep 11 17:13:53.655133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:53.655098    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerStarted","Data":"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"}
Sep 11 17:13:53.678558 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:53.678503    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podStartSLOduration=2.794039193 podStartE2EDuration="5.678486398s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.884447205s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:53.676861218 +0000 UTC m=+3424.451561798" watchObservedRunningTime="2026-09-11 17:13:53.678486398 +0000 UTC m=+3424.453186973"
Sep 11 17:13:54.112889 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:54.112847    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:59.113197 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:59.113144    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:13:59.114650 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:13:59.114607    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:02.370937 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:02.370898    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:14:09.113339 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:09.113290    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:19.113047 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:19.113000    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:22.233524 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:22.233484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:14:22.333269 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:22.333233    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:14:29.113361 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:29.113307    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:39.113819 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:39.113772    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:49.113658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:49.113593    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:14:59.113109 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:14:59.113059    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:15:09.123070 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:09.123027    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:15:09.131029 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:09.131004    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:15:10.493288 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:10.493254    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:15:10.498667 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:10.498639    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:10.559860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:10.559822    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:12.185154 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:12.185122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:12.281569 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:12.281487    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:12.447218 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:12.447181    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:12.543618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:12.543580    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:15.910298 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:15.910260    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:18.713784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:18.713747    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:22.318292 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:22.318258    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:22.698528 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:22.698488    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:22.798536 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:22.798498    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:24.633641 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:24.633587    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:24.700065 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:24.700034    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:15:26.058567 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.058529    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" containerID="cri-o://e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c" gracePeriod=45
Sep 11 17:15:26.124242 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.124204    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 11 17:15:26.601835 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.601811    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:15:26.687940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.687856    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.687940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.687903    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.687940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.687925    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688209 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.687961    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl6sp\" (UniqueName: \"kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688209 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688004    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688209 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688067    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688209 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688120    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688209 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688150    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm\") pod \"30719dfc-e23f-41e8-bf0c-20466335b7f1\" (UID: \"30719dfc-e23f-41e8-bf0c-20466335b7f1\") "
Sep 11 17:15:26.688457 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688281    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:15:26.688496 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688449    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:15:26.688743 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688711    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:15:26.688858 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.688813    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home" (OuterVolumeSpecName: "home") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:15:26.690388 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.690361    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm" (OuterVolumeSpecName: "dshm") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:15:26.690388 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.690353    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp" (OuterVolumeSpecName: "kube-api-access-xl6sp") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "kube-api-access-xl6sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:15:26.690875 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.690847    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:15:26.710637 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.710594    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30719dfc-e23f-41e8-bf0c-20466335b7f1" (UID: "30719dfc-e23f-41e8-bf0c-20466335b7f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:15:26.788974 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.788946    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.788974 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.788971    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.788987    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.788999    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30719dfc-e23f-41e8-bf0c-20466335b7f1-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.789035    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.789050    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30719dfc-e23f-41e8-bf0c-20466335b7f1-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.789063    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6sp\" (UniqueName: \"kubernetes.io/projected/30719dfc-e23f-41e8-bf0c-20466335b7f1-kube-api-access-xl6sp\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:26.789133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:26.789075    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30719dfc-e23f-41e8-bf0c-20466335b7f1-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:15:27.118793 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.118757    2550 generic.go:356] "Generic (PLEG): container finished" podID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerID="e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c" exitCode=0
Sep 11 17:15:27.119307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.118803    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"
Sep 11 17:15:27.119307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.118813    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerDied","Data":"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"}
Sep 11 17:15:27.119307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.118870    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c" event={"ID":"30719dfc-e23f-41e8-bf0c-20466335b7f1","Type":"ContainerDied","Data":"334f595236e83da19915063f78c2b2a4dd81678747447dfa1cf190505ed597fb"}
Sep 11 17:15:27.119307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.118901    2550 scope.go:117] "RemoveContainer" containerID="e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"
Sep 11 17:15:27.128470 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.128449    2550 scope.go:117] "RemoveContainer" containerID="655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"
Sep 11 17:15:27.136857 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.136837    2550 scope.go:117] "RemoveContainer" containerID="e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"
Sep 11 17:15:27.137139 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:15:27.137120    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c\": container with ID starting with e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c not found: ID does not exist" containerID="e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"
Sep 11 17:15:27.137193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.137148    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c"} err="failed to get container status \"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c\": rpc error: code = NotFound desc = could not find container \"e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c\": container with ID starting with e2478977660c5331aa44c7e97ce595721cd3f1c46ba00c6b03c03dfb8911208c not found: ID does not exist"
Sep 11 17:15:27.137193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.137167    2550 scope.go:117] "RemoveContainer" containerID="655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"
Sep 11 17:15:27.137457 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:15:27.137436    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c\": container with ID starting with 655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c not found: ID does not exist" containerID="655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"
Sep 11 17:15:27.137500 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.137467    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c"} err="failed to get container status \"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c\": rpc error: code = NotFound desc = could not find container \"655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c\": container with ID starting with 655332d5e2be740a344c2135c3f32dfc2ad5aa88b0d888322540324fb82d598c not found: ID does not exist"
Sep 11 17:15:27.142354 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.142329    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:15:27.146645 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.146606    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7585fc7f7f-d8b9c"]
Sep 11 17:15:27.688513 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:15:27.688481    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" path="/var/lib/kubelet/pods/30719dfc-e23f-41e8-bf0c-20466335b7f1/volumes"
Sep 11 17:16:00.605197 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:00.605151    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:02.515673 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:02.515635    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:02.870796 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:02.870699    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:02.971939 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:02.971904    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:03.074719 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:03.074684    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:06.834322 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:06.834275    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:08.532390 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:08.532351    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:08.632428 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:08.632392    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:12.409490 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:12.409449    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:12.470775 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:12.470737    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:26.270555 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270521    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:26.271101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270898    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main"
Sep 11 17:16:26.271101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270911    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main"
Sep 11 17:16:26.271101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270930    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="storage-initializer"
Sep 11 17:16:26.271101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270936    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="storage-initializer"
Sep 11 17:16:26.271101 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.270996    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="30719dfc-e23f-41e8-bf0c-20466335b7f1" containerName="main"
Sep 11 17:16:26.273743 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.273725    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.276722 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.276694    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 11 17:16:26.276858 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.276722    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 11 17:16:26.276858 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.276738    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 11 17:16:26.276858 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.276830    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 11 17:16:26.277032 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.276893    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-jzs9d\""
Sep 11 17:16:26.288422 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.288398    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:26.298121 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.298101    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:26.395222 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395195    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395246    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprwp\" (UniqueName: \"kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395305    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395324    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395369    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.395569 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.395399    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496003 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.495967    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprwp\" (UniqueName: \"kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496060    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496090    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496117    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496166    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496461 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496520 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496475    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496520 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496478    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496811 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.496916 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.496812    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.498378 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.498355    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.498745 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.498730    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.504341 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.504321    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprwp\" (UniqueName: \"kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp\") pod \"e2e-del-blocked-svc-kserve-7ddd564dcb-c4558\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.511040 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.511008    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:26.584499 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.584439    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:26.858681 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:26.858652    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:26.860664 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:16:26.860605    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f70466_73d4_4efb_b0f6_094ce6fcded8.slice/crio-99712cea02bdecc6e39f43930789b7c96269b92e38319ea438d948cbcd45e5bd WatchSource:0}: Error finding container 99712cea02bdecc6e39f43930789b7c96269b92e38319ea438d948cbcd45e5bd: Status 404 returned error can't find the container with id 99712cea02bdecc6e39f43930789b7c96269b92e38319ea438d948cbcd45e5bd
Sep 11 17:16:27.399223 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:27.399186    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558" event={"ID":"61f70466-73d4-4efb-b0f6-094ce6fcded8","Type":"ContainerStarted","Data":"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"}
Sep 11 17:16:27.399717 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:27.399245    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558" event={"ID":"61f70466-73d4-4efb-b0f6-094ce6fcded8","Type":"ContainerStarted","Data":"99712cea02bdecc6e39f43930789b7c96269b92e38319ea438d948cbcd45e5bd"}
Sep 11 17:16:30.201617 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:30.201583    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:30.202128 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:30.201814    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558" podUID="61f70466-73d4-4efb-b0f6-094ce6fcded8" containerName="storage-initializer" containerID="cri-o://f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7" gracePeriod=60
Sep 11 17:16:30.213458 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:30.213423    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:30.275529 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:30.275484    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:31.413932 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:31.413893    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:31.570927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:31.570890    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:31.671902 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:31.671807    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:32.647115 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.647092    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:32.760070 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760040    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nprwp\" (UniqueName: \"kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760261 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760083    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760261 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760113    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760261 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760237    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760268    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760298    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760350    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:16:32.760433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760361    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache" (OuterVolumeSpecName: "model-cache") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:16:32.760433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760365    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location\") pod \"61f70466-73d4-4efb-b0f6-094ce6fcded8\" (UID: \"61f70466-73d4-4efb-b0f6-094ce6fcded8\") "
Sep 11 17:16:32.760655 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760507    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home" (OuterVolumeSpecName: "home") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:16:32.760815 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760781    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.760815 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760814    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.761014 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.760823    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.762320 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.762293    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:16:32.762407 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.762380    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp" (OuterVolumeSpecName: "kube-api-access-nprwp") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "kube-api-access-nprwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:16:32.762758 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.762737    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm" (OuterVolumeSpecName: "dshm") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:16:32.862250 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.862225    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprwp\" (UniqueName: \"kubernetes.io/projected/61f70466-73d4-4efb-b0f6-094ce6fcded8-kube-api-access-nprwp\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.862250 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.862249    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f70466-73d4-4efb-b0f6-094ce6fcded8-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.862380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.862258    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:32.939419 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.939388    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61f70466-73d4-4efb-b0f6-094ce6fcded8" (UID: "61f70466-73d4-4efb-b0f6-094ce6fcded8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:16:32.963143 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:32.963119    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61f70466-73d4-4efb-b0f6-094ce6fcded8-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:16:33.437275 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.437235    2550 generic.go:356] "Generic (PLEG): container finished" podID="61f70466-73d4-4efb-b0f6-094ce6fcded8" containerID="f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7" exitCode=0
Sep 11 17:16:33.437456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.437283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558" event={"ID":"61f70466-73d4-4efb-b0f6-094ce6fcded8","Type":"ContainerDied","Data":"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"}
Sep 11 17:16:33.437456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.437324    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558" event={"ID":"61f70466-73d4-4efb-b0f6-094ce6fcded8","Type":"ContainerDied","Data":"99712cea02bdecc6e39f43930789b7c96269b92e38319ea438d948cbcd45e5bd"}
Sep 11 17:16:33.437456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.437349    2550 scope.go:117] "RemoveContainer" containerID="f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"
Sep 11 17:16:33.437456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.437289    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"
Sep 11 17:16:33.447134 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.447112    2550 scope.go:117] "RemoveContainer" containerID="f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"
Sep 11 17:16:33.447396 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:16:33.447373    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7\": container with ID starting with f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7 not found: ID does not exist" containerID="f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"
Sep 11 17:16:33.447442 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.447406    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7"} err="failed to get container status \"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7\": rpc error: code = NotFound desc = could not find container \"f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7\": container with ID starting with f982179552d2aae3251f245133e7d189d3d8fcc276272407179c114dfba5fbd7 not found: ID does not exist"
Sep 11 17:16:33.477938 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.477907    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:33.480660 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.480635    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7ddd564dcb-c4558"]
Sep 11 17:16:33.687423 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:33.687349    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f70466-73d4-4efb-b0f6-094ce6fcded8" path="/var/lib/kubelet/pods/61f70466-73d4-4efb-b0f6-094ce6fcded8/volumes"
Sep 11 17:16:35.818613 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:35.818576    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:35.922075 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:35.922038    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:36.019383 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:36.019351    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:36.495995 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:36.495957    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:40.180006 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:40.179964    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:40.366286 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:40.366247    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:40.466213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:40.466180    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:41.831567 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:41.831532    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:41.931551 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:41.931517    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:46.000393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:46.000317    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:16:46.099694 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:16:46.099659    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:03.571461 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.571426    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:03.572011 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.571990    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f70466-73d4-4efb-b0f6-094ce6fcded8" containerName="storage-initializer"
Sep 11 17:17:03.572065 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.572015    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f70466-73d4-4efb-b0f6-094ce6fcded8" containerName="storage-initializer"
Sep 11 17:17:03.572111 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.572102    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f70466-73d4-4efb-b0f6-094ce6fcded8" containerName="storage-initializer"
Sep 11 17:17:03.576600 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.576578    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.579606 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.579583    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 11 17:17:03.579736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.579671    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-f2s8k\""
Sep 11 17:17:03.579736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.579701    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 11 17:17:03.579833 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.579745    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 11 17:17:03.585813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.585788    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:03.605524 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.605503    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:03.620697 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.620675    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g64x2\" (UniqueName: \"kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.620778 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.620717    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.721722 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.721690    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g64x2\" (UniqueName: \"kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.721886 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.721746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.722115 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.722097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.731500 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.731470    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g64x2\" (UniqueName: \"kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2\") pod \"e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.779463 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.779430    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:03.783469 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.783446    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:03.786401 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.786379    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-qdtd6\""
Sep 11 17:17:03.786532 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.786462    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 11 17:17:03.792332 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.792312    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:03.811099 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.811079    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:03.887321 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.887257    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:03.924771 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.924733    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9p4\" (UniqueName: \"kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:03.925118 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:03.925058    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.026506 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.026471    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9p4\" (UniqueName: \"kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.026713 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.026541    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.028955 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.028929    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.034407 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.034383    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9p4\" (UniqueName: \"kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.095481 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.095458    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.160500 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.160461    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:04.162675 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:17:04.162649    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48e2e7f_6872_41d5_a32c_8369140f73c1.slice/crio-d609823396251b587db1b11613662134e38643108cd71075de05843fa00e230c WatchSource:0}: Error finding container d609823396251b587db1b11613662134e38643108cd71075de05843fa00e230c: Status 404 returned error can't find the container with id d609823396251b587db1b11613662134e38643108cd71075de05843fa00e230c
Sep 11 17:17:04.366038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.365936    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:04.368276 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:17:04.368236    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7572b06_1ee0_4998_bd17_62c6dfc57e57.slice/crio-fde79b7bf48534516817220277d60445ec1ee3cea77f8d78b013c7ac4edf4e83 WatchSource:0}: Error finding container fde79b7bf48534516817220277d60445ec1ee3cea77f8d78b013c7ac4edf4e83: Status 404 returned error can't find the container with id fde79b7bf48534516817220277d60445ec1ee3cea77f8d78b013c7ac4edf4e83
Sep 11 17:17:04.579709 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.579669    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" event={"ID":"f7572b06-1ee0-4998-bd17-62c6dfc57e57","Type":"ContainerStarted","Data":"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"}
Sep 11 17:17:04.580199 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.579737    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" event={"ID":"f7572b06-1ee0-4998-bd17-62c6dfc57e57","Type":"ContainerStarted","Data":"fde79b7bf48534516817220277d60445ec1ee3cea77f8d78b013c7ac4edf4e83"}
Sep 11 17:17:04.580199 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.579766    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:04.581253 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.581229    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z" event={"ID":"c48e2e7f-6872-41d5-a32c-8369140f73c1","Type":"ContainerStarted","Data":"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"}
Sep 11 17:17:04.581376 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.581269    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z" event={"ID":"c48e2e7f-6872-41d5-a32c-8369140f73c1","Type":"ContainerStarted","Data":"d609823396251b587db1b11613662134e38643108cd71075de05843fa00e230c"}
Sep 11 17:17:04.619633 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:04.619574    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" podStartSLOduration=1.6195618920000001 podStartE2EDuration="1.619561892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:17:04.616060624 +0000 UTC m=+3615.390761201" watchObservedRunningTime="2026-09-11 17:17:04.619561892 +0000 UTC m=+3615.394262471"
Sep 11 17:17:07.015787 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:07.015745    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:07.017963 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:07.017925    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:07.018159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:07.018117    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z" podUID="c48e2e7f-6872-41d5-a32c-8369140f73c1" containerName="storage-initializer" containerID="cri-o://495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5" gracePeriod=30
Sep 11 17:17:07.036029 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:07.035994    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:07.102460 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:07.102425    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:08.931423 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:08.931371    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:09.027740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.027697    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:09.762374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.762350    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:09.886385 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.886293    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location\") pod \"c48e2e7f-6872-41d5-a32c-8369140f73c1\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") "
Sep 11 17:17:09.886385 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.886351    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g64x2\" (UniqueName: \"kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2\") pod \"c48e2e7f-6872-41d5-a32c-8369140f73c1\" (UID: \"c48e2e7f-6872-41d5-a32c-8369140f73c1\") "
Sep 11 17:17:09.888603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.888570    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2" (OuterVolumeSpecName: "kube-api-access-g64x2") pod "c48e2e7f-6872-41d5-a32c-8369140f73c1" (UID: "c48e2e7f-6872-41d5-a32c-8369140f73c1"). InnerVolumeSpecName "kube-api-access-g64x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:17:09.959433 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.959391    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c48e2e7f-6872-41d5-a32c-8369140f73c1" (UID: "c48e2e7f-6872-41d5-a32c-8369140f73c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:17:09.987559 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.987532    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g64x2\" (UniqueName: \"kubernetes.io/projected/c48e2e7f-6872-41d5-a32c-8369140f73c1-kube-api-access-g64x2\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:17:09.987559 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:09.987557    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c48e2e7f-6872-41d5-a32c-8369140f73c1-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:17:10.615481 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.615440    2550 generic.go:356] "Generic (PLEG): container finished" podID="c48e2e7f-6872-41d5-a32c-8369140f73c1" containerID="495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5" exitCode=0
Sep 11 17:17:10.615692 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.615485    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"
Sep 11 17:17:10.615692 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.615524    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z" event={"ID":"c48e2e7f-6872-41d5-a32c-8369140f73c1","Type":"ContainerDied","Data":"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"}
Sep 11 17:17:10.615692 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.615577    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z" event={"ID":"c48e2e7f-6872-41d5-a32c-8369140f73c1","Type":"ContainerDied","Data":"d609823396251b587db1b11613662134e38643108cd71075de05843fa00e230c"}
Sep 11 17:17:10.615692 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.615599    2550 scope.go:117] "RemoveContainer" containerID="495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"
Sep 11 17:17:10.624980 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.624965    2550 scope.go:117] "RemoveContainer" containerID="495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"
Sep 11 17:17:10.625295 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:17:10.625270    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5\": container with ID starting with 495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5 not found: ID does not exist" containerID="495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"
Sep 11 17:17:10.625374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.625306    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5"} err="failed to get container status \"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5\": rpc error: code = NotFound desc = could not find container \"495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5\": container with ID starting with 495871a9df72b0b120dcacd0ec6828954faa3210a18b393f08314aeeb9faedf5 not found: ID does not exist"
Sep 11 17:17:10.652372 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.652349    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:10.656364 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:10.656339    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-55fbbc9f6f-wx66z"]
Sep 11 17:17:11.686926 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:11.686888    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48e2e7f-6872-41d5-a32c-8369140f73c1" path="/var/lib/kubelet/pods/c48e2e7f-6872-41d5-a32c-8369140f73c1/volumes"
Sep 11 17:17:13.993988 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:13.993944    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:22.037406 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:22.037355    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main" containerID="cri-o://a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7" gracePeriod=45
Sep 11 17:17:28.012187 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:28.012151    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:28.201153 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:28.201112    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:37.019080 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:37.019037    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:17:44.159152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:44.159114    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:47.017789 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:47.017749    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:17:52.287603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.287580    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:52.347203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.347130    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs\") pod \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") "
Sep 11 17:17:52.347203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.347170    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt9p4\" (UniqueName: \"kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4\") pod \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\" (UID: \"f7572b06-1ee0-4998-bd17-62c6dfc57e57\") "
Sep 11 17:17:52.349162 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.349133    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7572b06-1ee0-4998-bd17-62c6dfc57e57" (UID: "f7572b06-1ee0-4998-bd17-62c6dfc57e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:17:52.349276 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.349196    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4" (OuterVolumeSpecName: "kube-api-access-rt9p4") pod "f7572b06-1ee0-4998-bd17-62c6dfc57e57" (UID: "f7572b06-1ee0-4998-bd17-62c6dfc57e57"). InnerVolumeSpecName "kube-api-access-rt9p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:17:52.399326 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.399295    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:52.448615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.448589    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7572b06-1ee0-4998-bd17-62c6dfc57e57-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:17:52.448615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.448614    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9p4\" (UniqueName: \"kubernetes.io/projected/f7572b06-1ee0-4998-bd17-62c6dfc57e57-kube-api-access-rt9p4\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:17:52.805928 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.805895    2550 generic.go:356] "Generic (PLEG): container finished" podID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerID="a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7" exitCode=0
Sep 11 17:17:52.806097 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.805939    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"
Sep 11 17:17:52.806097 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.805951    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" event={"ID":"f7572b06-1ee0-4998-bd17-62c6dfc57e57","Type":"ContainerDied","Data":"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"}
Sep 11 17:17:52.806097 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.805989    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk" event={"ID":"f7572b06-1ee0-4998-bd17-62c6dfc57e57","Type":"ContainerDied","Data":"fde79b7bf48534516817220277d60445ec1ee3cea77f8d78b013c7ac4edf4e83"}
Sep 11 17:17:52.806097 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.806011    2550 scope.go:117] "RemoveContainer" containerID="a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"
Sep 11 17:17:52.814739 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.814716    2550 scope.go:117] "RemoveContainer" containerID="a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"
Sep 11 17:17:52.815002 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:17:52.814983    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7\": container with ID starting with a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7 not found: ID does not exist" containerID="a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"
Sep 11 17:17:52.815042 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.815013    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7"} err="failed to get container status \"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7\": rpc error: code = NotFound desc = could not find container \"a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7\": container with ID starting with a6fd05288aafafa8b5138e017f8c6da8b9ff8d1d806edc575665d78d456b0da7 not found: ID does not exist"
Sep 11 17:17:52.827291 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.827269    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:52.830129 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:52.830108    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddcwksk"]
Sep 11 17:17:53.686588 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:53.686550    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" path="/var/lib/kubelet/pods/f7572b06-1ee0-4998-bd17-62c6dfc57e57/volumes"
Sep 11 17:17:54.297461 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:54.297426    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:17:54.352723 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:17:54.352605    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:21.799973 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:21.799887    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:41.193439 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:41.193396    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:41.287522 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:41.287491    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:45.612909 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.612874    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:18:45.613338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613304    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48e2e7f-6872-41d5-a32c-8369140f73c1" containerName="storage-initializer"
Sep 11 17:18:45.613338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613318    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48e2e7f-6872-41d5-a32c-8369140f73c1" containerName="storage-initializer"
Sep 11 17:18:45.613429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613344    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main"
Sep 11 17:18:45.613429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613350    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main"
Sep 11 17:18:45.613429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613406    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48e2e7f-6872-41d5-a32c-8369140f73c1" containerName="storage-initializer"
Sep 11 17:18:45.613429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.613426    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7572b06-1ee0-4998-bd17-62c6dfc57e57" containerName="main"
Sep 11 17:18:45.616732 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.616713    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.619393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.619362    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 11 17:18:45.619797 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.619774    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4zpqq\""
Sep 11 17:18:45.619928 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.619904    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:18:45.620283 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.620262    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 11 17:18:45.620578 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.620556    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 11 17:18:45.620707 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.620558    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 11 17:18:45.620707 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.620643    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-grhhj\""
Sep 11 17:18:45.630235 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.630213    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:18:45.679527 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.679497    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:45.685213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685190    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6v6\" (UniqueName: \"kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685229    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685290    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685324    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685375 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685368    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.685754 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.685429    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.786945 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.786912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6v6\" (UniqueName: \"kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787124 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.786957    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787124 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.786989    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787124 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787017    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787135    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787203    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787370 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787230    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787599 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787410    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787599 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787428    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787611    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787643    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.787855 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.787835    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.789871 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.789842    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.790127 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.790108    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.794880 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.794855    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6v6\" (UniqueName: \"kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.929203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.929129    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:45.957137 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:45.957103    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:46.212943 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:46.212918    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:18:46.214637 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:18:46.214574    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091c347d_009c_4e77_a34f_60f7e4daadf4.slice/crio-8d76a72fa31222c37ad35488c61034a870725c11dcb075d12ae44e944dcc5488 WatchSource:0}: Error finding container 8d76a72fa31222c37ad35488c61034a870725c11dcb075d12ae44e944dcc5488: Status 404 returned error can't find the container with id 8d76a72fa31222c37ad35488c61034a870725c11dcb075d12ae44e944dcc5488
Sep 11 17:18:46.214815 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:46.214785    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:47.061663 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:47.061617    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerStarted","Data":"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"}
Sep 11 17:18:47.062069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:47.061687    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerStarted","Data":"8d76a72fa31222c37ad35488c61034a870725c11dcb075d12ae44e944dcc5488"}
Sep 11 17:18:47.062069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:47.061755    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:48.069889 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:48.069828    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerStarted","Data":"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"}
Sep 11 17:18:50.081899 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:50.081859    2550 generic.go:356] "Generic (PLEG): container finished" podID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerID="40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053" exitCode=0
Sep 11 17:18:50.082359 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:50.081934    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerDied","Data":"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"}
Sep 11 17:18:51.089499 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:51.089465    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerStarted","Data":"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"}
Sep 11 17:18:51.115205 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:51.115145    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podStartSLOduration=3.864521993 podStartE2EDuration="6.115123909s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.250601916s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:18:51.112586048 +0000 UTC m=+3721.887286660" watchObservedRunningTime="2026-09-11 17:18:51.115123909 +0000 UTC m=+3721.889824489"
Sep 11 17:18:52.389919 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:52.389883    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:18:55.929429 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:55.929391    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:55.929921 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:55.929455    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:18:55.930789 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:18:55.930764    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:02.273548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:02.273499    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:03.746378 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:03.746333    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:04.672766 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:04.672730    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:05.929740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:05.929695    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:05.943447 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:05.943420    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:19:10.120993 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:10.120955    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:11.051120 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:11.051089    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:15.930318 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:15.930268    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:25.165967 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:25.165929    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:25.930459 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:25.930412    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:30.480006 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:30.479954    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:30.535908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:30.535868    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:32.716193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:32.716159    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:32.769681 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:32.769649    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:35.929988 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:35.929930    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:38.890198 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.890160    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:19:38.893834 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.893808    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.896607 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896571    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:19:38.896800 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896689    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 11 17:19:38.896800 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896750    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 11 17:19:38.896800 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896696    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 11 17:19:38.897028 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896755    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6xqjj\""
Sep 11 17:19:38.897028 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.896689    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 11 17:19:38.904323 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.904299    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:19:38.906738 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.906852 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906754    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.906852 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906779    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.906933 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwwh\" (UniqueName: \"kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.906933 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906905    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.906933 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906923    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.907051 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.906945    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.907093 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.907067    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:38.917730 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:38.917708    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:39.008328 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008286    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwwh\" (UniqueName: \"kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008328 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008328    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008542 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008502    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008586 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008548    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008641 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008610    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008685 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008733 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008700    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.008733 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.008724    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.009119 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.009089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.009289 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.009165    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.009492 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.009463    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.009879 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.009844    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.009996 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.009950    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.011115 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.011096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.011214 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.011198    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.021036 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.021016    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwwh\" (UniqueName: \"kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh\") pod \"stop-feature-test-kserve-7468ff9996-5476v\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.035994 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.035965    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:39.206453 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.206427    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:39.485079 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:39.485049    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:19:39.485670 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:19:39.485644    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a562f4d_defd_469f_bf0c_a696d9487f37.slice/crio-0d7b093b77d51103455dd44208318fa0ba32c5d1fe814af620b0a06720648a53 WatchSource:0}: Error finding container 0d7b093b77d51103455dd44208318fa0ba32c5d1fe814af620b0a06720648a53: Status 404 returned error can't find the container with id 0d7b093b77d51103455dd44208318fa0ba32c5d1fe814af620b0a06720648a53
Sep 11 17:19:40.340665 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:40.340603    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerStarted","Data":"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"}
Sep 11 17:19:40.341054 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:40.340699    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerStarted","Data":"0d7b093b77d51103455dd44208318fa0ba32c5d1fe814af620b0a06720648a53"}
Sep 11 17:19:42.352136 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:42.352050    2550 generic.go:356] "Generic (PLEG): container finished" podID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerID="18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21" exitCode=0
Sep 11 17:19:42.352136 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:42.352120    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerDied","Data":"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"}
Sep 11 17:19:43.359986 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:43.359945    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerStarted","Data":"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"}
Sep 11 17:19:43.386363 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:43.386304    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podStartSLOduration=3.032658438 podStartE2EDuration="5.38628711s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.353628672s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:43.383232026 +0000 UTC m=+3774.157932634" watchObservedRunningTime="2026-09-11 17:19:43.38628711 +0000 UTC m=+3774.160987687"
Sep 11 17:19:44.206728 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:44.206692    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:45.930158 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:45.930106    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:49.207437 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:49.207402    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:19:49.208980 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:49.208941    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:19:55.930412 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:55.930361    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:19:59.201389 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:59.201339    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:19:59.207302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:19:59.207271    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:03.478562 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:03.478524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:03.556697 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:03.556662    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:05.930451 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:05.930399    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:20:09.207171 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:09.207115    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:12.236414 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:12.236367    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:15.930182 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:15.930123    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:20:19.207516 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:19.207472    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:25.929645 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:25.929582    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:20:29.207640 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:29.207582    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:35.930067 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:35.930010    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Sep 11 17:20:39.207180 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:39.207130    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:45.944845 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:45.944809    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:20:45.968152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:45.968126    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:20:49.207018 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:49.206981    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:20:49.396477 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:49.396436    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:56.424203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:56.424155    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:56.434530 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:56.434491    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:20:56.487025 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:56.486981    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:20:59.206846 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:20:59.206801    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:21:09.206963 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:09.206916    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:21:11.462589 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:11.462551    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main" containerID="cri-o://0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf" gracePeriod=45
Sep 11 17:21:12.307733 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.307682    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="llm-d-routing-sidecar" containerID="cri-o://63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f" gracePeriod=45
Sep 11 17:21:12.566867 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.566843    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:21:12.613858 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.613827    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.613873    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.613896    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.613920    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.613958    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614305 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.614169    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.614305 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.614229    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d6v6\" (UniqueName: \"kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614305 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.614266    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.614557 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.614517    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home" (OuterVolumeSpecName: "home") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.614725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.614638    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert\") pod \"091c347d-009c-4e77-a34f-60f7e4daadf4\" (UID: \"091c347d-009c-4e77-a34f-60f7e4daadf4\") "
Sep 11 17:21:12.615076 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.615015    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.615076 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.615041    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.615323 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.615296    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:12.616774 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.616710    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm" (OuterVolumeSpecName: "dshm") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.616864 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.616813    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.616864 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.616831    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6" (OuterVolumeSpecName: "kube-api-access-8d6v6") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "kube-api-access-8d6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:12.633248 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.633224    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.640130 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.640107    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "091c347d-009c-4e77-a34f-60f7e4daadf4" (UID: "091c347d-009c-4e77-a34f-60f7e4daadf4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.715807 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715770    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c347d-009c-4e77-a34f-60f7e4daadf4-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.715807 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715804    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.716015 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715817    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.716015 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715830    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6v6\" (UniqueName: \"kubernetes.io/projected/091c347d-009c-4e77-a34f-60f7e4daadf4-kube-api-access-8d6v6\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.716015 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715844    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091c347d-009c-4e77-a34f-60f7e4daadf4-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.716015 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.715857    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/091c347d-009c-4e77-a34f-60f7e4daadf4-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.848634 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848534    2550 generic.go:356] "Generic (PLEG): container finished" podID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerID="0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf" exitCode=0
Sep 11 17:21:12.848634 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848560    2550 generic.go:356] "Generic (PLEG): container finished" podID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerID="63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f" exitCode=0
Sep 11 17:21:12.848634 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848609    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"
Sep 11 17:21:12.848860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848613    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerDied","Data":"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"}
Sep 11 17:21:12.848860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848688    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerDied","Data":"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"}
Sep 11 17:21:12.848860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848707    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5" event={"ID":"091c347d-009c-4e77-a34f-60f7e4daadf4","Type":"ContainerDied","Data":"8d76a72fa31222c37ad35488c61034a870725c11dcb075d12ae44e944dcc5488"}
Sep 11 17:21:12.848860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.848730    2550 scope.go:117] "RemoveContainer" containerID="0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"
Sep 11 17:21:12.857548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.857480    2550 scope.go:117] "RemoveContainer" containerID="40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"
Sep 11 17:21:12.866485 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.866467    2550 scope.go:117] "RemoveContainer" containerID="63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"
Sep 11 17:21:12.874331 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.874128    2550 scope.go:117] "RemoveContainer" containerID="0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"
Sep 11 17:21:12.874515 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:21:12.874496    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf\": container with ID starting with 0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf not found: ID does not exist" containerID="0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"
Sep 11 17:21:12.874588 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.874524    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"} err="failed to get container status \"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf\": rpc error: code = NotFound desc = could not find container \"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf\": container with ID starting with 0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf not found: ID does not exist"
Sep 11 17:21:12.874588 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.874546    2550 scope.go:117] "RemoveContainer" containerID="40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"
Sep 11 17:21:12.874869 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:21:12.874841    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053\": container with ID starting with 40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053 not found: ID does not exist" containerID="40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"
Sep 11 17:21:12.874918 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.874877    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"} err="failed to get container status \"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053\": rpc error: code = NotFound desc = could not find container \"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053\": container with ID starting with 40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053 not found: ID does not exist"
Sep 11 17:21:12.874918 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.874894    2550 scope.go:117] "RemoveContainer" containerID="63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"
Sep 11 17:21:12.875162 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:21:12.875145    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f\": container with ID starting with 63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f not found: ID does not exist" containerID="63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"
Sep 11 17:21:12.875241 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875172    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"} err="failed to get container status \"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f\": rpc error: code = NotFound desc = could not find container \"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f\": container with ID starting with 63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f not found: ID does not exist"
Sep 11 17:21:12.875241 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875186    2550 scope.go:117] "RemoveContainer" containerID="0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"
Sep 11 17:21:12.875462 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875443    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf"} err="failed to get container status \"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf\": rpc error: code = NotFound desc = could not find container \"0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf\": container with ID starting with 0c0911e9cb1ec86377ae884a4db2c6f183d11ca4408f2644a728d270a976cacf not found: ID does not exist"
Sep 11 17:21:12.875515 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875463    2550 scope.go:117] "RemoveContainer" containerID="40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"
Sep 11 17:21:12.875824 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875795    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053"} err="failed to get container status \"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053\": rpc error: code = NotFound desc = could not find container \"40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053\": container with ID starting with 40c7831a34b12619e1014f6cde59e0bf58821d5ddeaec5c7b26f99e7e2f0d053 not found: ID does not exist"
Sep 11 17:21:12.875900 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.875825    2550 scope.go:117] "RemoveContainer" containerID="63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"
Sep 11 17:21:12.876103 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.876083    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f"} err="failed to get container status \"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f\": rpc error: code = NotFound desc = could not find container \"63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f\": container with ID starting with 63788201e8fe2beb3dedc979b6c475c3c233c66f2829ae550230a10483bdb48f not found: ID does not exist"
Sep 11 17:21:12.877548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.877526    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:21:12.882798 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:12.882775    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-64bbd84dc42nbf5"]
Sep 11 17:21:13.688514 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:13.688474    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" path="/var/lib/kubelet/pods/091c347d-009c-4e77-a34f-60f7e4daadf4/volumes"
Sep 11 17:21:19.206918 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:19.206873    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:21:29.207696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:29.207649    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:21:39.217452 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:39.217415    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:21:39.225510 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:39.225485    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:21:40.543680 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:40.543614    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:40.544218 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:40.543696    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:21:41.899396 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:41.899364    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:46.577838 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:46.577802    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:46.676698 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:46.676662    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.669469 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:52.669428    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:56.010606 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.010543    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" containerID="cri-o://7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528" gracePeriod=45
Sep 11 17:21:56.218559 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.218516    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 11 17:21:56.661576 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.661553    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:21:56.707530 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707500    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707691 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707691 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707563    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707691 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707582    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707893 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707737    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707893 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707811    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707893 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707852    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.707893 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707882    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svwwh\" (UniqueName: \"kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh\") pod \"8a562f4d-defd-469f-bf0c-a696d9487f37\" (UID: \"8a562f4d-defd-469f-bf0c-a696d9487f37\") "
Sep 11 17:21:56.708119 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.707929    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:56.708187 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.708162    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:56.708296 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.708273    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home" (OuterVolumeSpecName: "home") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:56.708441 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.708372    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a562f4d-defd-469f-bf0c-a696d9487f37-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.708548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.708535    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.710746 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.710714    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:56.710857 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.710741    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm" (OuterVolumeSpecName: "dshm") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:56.710927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.710853    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh" (OuterVolumeSpecName: "kube-api-access-svwwh") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "kube-api-access-svwwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:56.723476 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.723452    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:56.735138 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.735084    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a562f4d-defd-469f-bf0c-a696d9487f37" (UID: "8a562f4d-defd-469f-bf0c-a696d9487f37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:56.809291 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809263    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.809291 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809288    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwwh\" (UniqueName: \"kubernetes.io/projected/8a562f4d-defd-469f-bf0c-a696d9487f37-kube-api-access-svwwh\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.809443 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809299    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.809443 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809308    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a562f4d-defd-469f-bf0c-a696d9487f37-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.809443 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809316    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:56.809443 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:56.809324    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a562f4d-defd-469f-bf0c-a696d9487f37-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:21:57.065233 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.065142    2550 generic.go:356] "Generic (PLEG): container finished" podID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerID="7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528" exitCode=0
Sep 11 17:21:57.065233 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.065204    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"
Sep 11 17:21:57.065790 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.065230    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerDied","Data":"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"}
Sep 11 17:21:57.065790 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.065302    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v" event={"ID":"8a562f4d-defd-469f-bf0c-a696d9487f37","Type":"ContainerDied","Data":"0d7b093b77d51103455dd44208318fa0ba32c5d1fe814af620b0a06720648a53"}
Sep 11 17:21:57.065790 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.065337    2550 scope.go:117] "RemoveContainer" containerID="7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"
Sep 11 17:21:57.074952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.074936    2550 scope.go:117] "RemoveContainer" containerID="18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"
Sep 11 17:21:57.084764 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.084742    2550 scope.go:117] "RemoveContainer" containerID="7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"
Sep 11 17:21:57.085074 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:21:57.085054    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528\": container with ID starting with 7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528 not found: ID does not exist" containerID="7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"
Sep 11 17:21:57.085132 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.085082    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528"} err="failed to get container status \"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528\": rpc error: code = NotFound desc = could not find container \"7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528\": container with ID starting with 7750519b711071597939aba244420124abea66a1959eae6f937e92a4afb2d528 not found: ID does not exist"
Sep 11 17:21:57.085132 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.085102    2550 scope.go:117] "RemoveContainer" containerID="18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"
Sep 11 17:21:57.085343 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:21:57.085323    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21\": container with ID starting with 18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21 not found: ID does not exist" containerID="18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"
Sep 11 17:21:57.085427 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.085346    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21"} err="failed to get container status \"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21\": rpc error: code = NotFound desc = could not find container \"18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21\": container with ID starting with 18a510a26e94514dd05cc20a391f51e92db191b0e5b89fed2ebf78f192aeda21 not found: ID does not exist"
Sep 11 17:21:57.088524 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.088498    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:21:57.092614 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.092592    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-5476v"]
Sep 11 17:21:57.688190 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:57.688156    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" path="/var/lib/kubelet/pods/8a562f4d-defd-469f-bf0c-a696d9487f37/volumes"
Sep 11 17:21:58.108499 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:58.108458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:58.207335 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:21:58.207300    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:05.653901 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.653871    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:22:05.654432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654365    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main"
Sep 11 17:22:05.654432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654388    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main"
Sep 11 17:22:05.654432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654407    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main"
Sep 11 17:22:05.654432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654416    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main"
Sep 11 17:22:05.654432 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654432    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="storage-initializer"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654442    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="storage-initializer"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654455    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="storage-initializer"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654464    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="storage-initializer"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654486    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="llm-d-routing-sidecar"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654495    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="llm-d-routing-sidecar"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654611    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="main"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654639    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a562f4d-defd-469f-bf0c-a696d9487f37" containerName="main"
Sep 11 17:22:05.654725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.654653    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="091c347d-009c-4e77-a34f-60f7e4daadf4" containerName="llm-d-routing-sidecar"
Sep 11 17:22:05.657940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.657920    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.660471 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660450    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 11 17:22:05.660586 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660451    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pmvlv\""
Sep 11 17:22:05.660882 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660860    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:22:05.661026 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660900    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 11 17:22:05.661026 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660954    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 11 17:22:05.661026 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.660985    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 11 17:22:05.669311 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.669285    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:22:05.698118 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.698094    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:05.784166 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784132    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784279 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784172    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784279 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784197    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784369 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784291    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784369 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784317    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784454 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784363    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784454 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784405    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.784454 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.784439    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.884938 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.884901    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.884938 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.884937    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885160 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885061    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885160 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885132    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885272 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885272 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885239    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885270    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885282    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885296    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885560 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885489    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885560 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885535    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885784 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885762    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.885854 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.885769    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.887394 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.887369    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.887670 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.887652    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.893252 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.893230    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q\") pod \"custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:05.902743 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.902721    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:05.969719 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:05.969681    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:06.249347 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:06.249323    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:22:06.251776 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:22:06.251742    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba3ea3_4415_4fc6_ad22_3b3bed789114.slice/crio-bebad5bcb0408b0c02b86c763d6f220600cb1395bbe147bc56f41b864a439642 WatchSource:0}: Error finding container bebad5bcb0408b0c02b86c763d6f220600cb1395bbe147bc56f41b864a439642: Status 404 returned error can't find the container with id bebad5bcb0408b0c02b86c763d6f220600cb1395bbe147bc56f41b864a439642
Sep 11 17:22:07.116386 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:07.116341    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerStarted","Data":"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"}
Sep 11 17:22:07.116896 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:07.116444    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerStarted","Data":"bebad5bcb0408b0c02b86c763d6f220600cb1395bbe147bc56f41b864a439642"}
Sep 11 17:22:09.128613 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:09.128570    2550 generic.go:356] "Generic (PLEG): container finished" podID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerID="86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246" exitCode=0
Sep 11 17:22:09.129098 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:09.128665    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerDied","Data":"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"}
Sep 11 17:22:10.136041 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:10.136002    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerStarted","Data":"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"}
Sep 11 17:22:10.158058 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:10.158010    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podStartSLOduration=2.876862017 podStartE2EDuration="5.157993991s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.281131974s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:10.156256755 +0000 UTC m=+3920.930957332" watchObservedRunningTime="2026-09-11 17:22:10.157993991 +0000 UTC m=+3920.932694568"
Sep 11 17:22:15.969879 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:15.969846    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:15.970343 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:15.969920    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:22:15.971472 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:15.971445    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:22:20.393915 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:20.393870    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:25.970445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:25.970409    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:22:30.404790 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:30.404693    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:30.504078 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:30.504039    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:30.608380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:30.608347    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:35.970916 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:35.970875    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:22:36.642157 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.642123    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:36.796693 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.796240    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:22:36.800881 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.800853    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.804075 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.804049    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 11 17:22:36.804307 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.804285    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 11 17:22:36.804693 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.804676    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-9gb8m\""
Sep 11 17:22:36.804780 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.804679    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:22:36.806048 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.806020    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 11 17:22:36.806159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.806070    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 11 17:22:36.807826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.807795    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:36.816669 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.816645    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:22:36.868714 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868677    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868840 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868732    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868840 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868840 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868797    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flcn\" (UniqueName: \"kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868972 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868860    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868972 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868884    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868972 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868907    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.868972 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.868931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969544 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969513    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969569    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969668    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969728    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flcn\" (UniqueName: \"kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969917 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969773    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969917 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969790    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.969917 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.969822    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970093 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970015    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970160 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970096    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970160 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970113    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970343 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970292    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970408    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.970581 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.970553    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.971988 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.971968    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.972548 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.972530    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:36.978999 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:36.978974    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flcn\" (UniqueName: \"kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn\") pod \"lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:37.112668 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:37.112610    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:37.391724 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:22:37.391693    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e903b5_5096_47fb_adfc_dc2e231939aa.slice/crio-4957cf92fb1471e1c82b42fd85d3ef74188751bffc1342ffe7ac943c2d01e5d0 WatchSource:0}: Error finding container 4957cf92fb1471e1c82b42fd85d3ef74188751bffc1342ffe7ac943c2d01e5d0: Status 404 returned error can't find the container with id 4957cf92fb1471e1c82b42fd85d3ef74188751bffc1342ffe7ac943c2d01e5d0
Sep 11 17:22:37.391872 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:37.391845    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:22:38.291422 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:38.291381    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerStarted","Data":"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"}
Sep 11 17:22:38.291898 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:38.291437    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerStarted","Data":"4957cf92fb1471e1c82b42fd85d3ef74188751bffc1342ffe7ac943c2d01e5d0"}
Sep 11 17:22:41.309175 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:41.309136    2550 generic.go:356] "Generic (PLEG): container finished" podID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerID="264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea" exitCode=0
Sep 11 17:22:41.309594 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:41.309211    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerDied","Data":"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"}
Sep 11 17:22:42.117492 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:42.117458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:42.182068 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:42.182025    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:42.316966 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:42.316883    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerStarted","Data":"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"}
Sep 11 17:22:42.343344 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:42.343294    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podStartSLOduration=3.133670752 podStartE2EDuration="6.343275685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.209604933s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:42.340642098 +0000 UTC m=+3953.115342675" watchObservedRunningTime="2026-09-11 17:22:42.343275685 +0000 UTC m=+3953.117976263"
Sep 11 17:22:43.113067 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:43.113026    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:45.970983 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:45.970944    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:22:47.113787 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:47.113677    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:22:47.115326 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:47.115297    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:22:55.970749 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:55.970701    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:22:57.113635 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:22:57.113586    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:05.970161 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:05.970116    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:07.113956 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:07.113917    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:15.970321 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:15.970279    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:17.113332 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:17.113288    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:25.970857 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:25.970811    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:27.113396 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:27.113353    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:35.970983 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:35.970934    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:37.113458 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:37.113400    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:40.948962 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:40.948925    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:45.970456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:45.970412    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:47.113331 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:47.113289    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:23:55.970813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:55.970770    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 11 17:23:57.123843 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:57.123811    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:23:57.131851 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:23:57.131823    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:24:05.979823 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:05.979784    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:24:05.987921 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:05.987894    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:24:18.148860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:18.148645    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:18.158077 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:18.158036    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:24:18.213899 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:18.213821    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:26.806732 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:26.806689    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7_34e903b5-5096-47fb-adfc-dc2e231939aa/storage-initializer/0.log"
Sep 11 17:24:26.818306 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:26.818277    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7_34e903b5-5096-47fb-adfc-dc2e231939aa/main/0.log"
Sep 11 17:24:27.977447 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:27.977385    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:24:27.994516 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:27.994473    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:33.186661 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.186603    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main" containerID="cri-o://4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa" gracePeriod=45
Sep 11 17:24:33.850420 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.850395    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:24:33.915811 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.915781    2550 generic.go:356] "Generic (PLEG): container finished" podID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerID="4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa" exitCode=0
Sep 11 17:24:33.915960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.915829    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"
Sep 11 17:24:33.915960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.915850    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerDied","Data":"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"}
Sep 11 17:24:33.915960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.915907    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8" event={"ID":"5eba3ea3-4415-4fc6-ad22-3b3bed789114","Type":"ContainerDied","Data":"bebad5bcb0408b0c02b86c763d6f220600cb1395bbe147bc56f41b864a439642"}
Sep 11 17:24:33.915960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.915933    2550 scope.go:117] "RemoveContainer" containerID="4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"
Sep 11 17:24:33.924604 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924586    2550 scope.go:117] "RemoveContainer" containerID="86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"
Sep 11 17:24:33.924797 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924777    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.924853 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924818    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.924898 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924851    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.924898 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924881    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.925003 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924936    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.925003 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.924971    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.925108 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.925032    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.925166 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.925119    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache\") pod \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\" (UID: \"5eba3ea3-4415-4fc6-ad22-3b3bed789114\") "
Sep 11 17:24:33.925394 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.925362    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:24:33.925771 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.925566    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home" (OuterVolumeSpecName: "home") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:33.925879 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.925814    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache" (OuterVolumeSpecName: "model-cache") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:33.927882 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.927859    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm" (OuterVolumeSpecName: "dshm") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:33.927985 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.927879    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:33.927985 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.927882    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q" (OuterVolumeSpecName: "kube-api-access-z9j2q") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "kube-api-access-z9j2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:24:33.938099 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.938035    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:33.944387 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.944363    2550 scope.go:117] "RemoveContainer" containerID="4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"
Sep 11 17:24:33.944707 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:24:33.944685    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa\": container with ID starting with 4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa not found: ID does not exist" containerID="4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"
Sep 11 17:24:33.944777 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.944716    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa"} err="failed to get container status \"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa\": rpc error: code = NotFound desc = could not find container \"4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa\": container with ID starting with 4659bdfb575eee801fe947dd580742411e64ce1907d5bf9f5d35c39ed50050fa not found: ID does not exist"
Sep 11 17:24:33.944777 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.944734    2550 scope.go:117] "RemoveContainer" containerID="86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"
Sep 11 17:24:33.945007 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:24:33.944989    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246\": container with ID starting with 86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246 not found: ID does not exist" containerID="86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"
Sep 11 17:24:33.945053 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.945013    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246"} err="failed to get container status \"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246\": rpc error: code = NotFound desc = could not find container \"86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246\": container with ID starting with 86d4d4435192b268ebc73fda9baee92d25392ef09ef9547052b5b4b2ce753246 not found: ID does not exist"
Sep 11 17:24:33.950377 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:33.950355    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5eba3ea3-4415-4fc6-ad22-3b3bed789114" (UID: "5eba3ea3-4415-4fc6-ad22-3b3bed789114"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.026495 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026470    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026495 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026491    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9j2q\" (UniqueName: \"kubernetes.io/projected/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kube-api-access-z9j2q\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026502    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026512    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026519    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eba3ea3-4415-4fc6-ad22-3b3bed789114-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026528    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026537    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eba3ea3-4415-4fc6-ad22-3b3bed789114-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.026658 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.026546    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba3ea3-4415-4fc6-ad22-3b3bed789114-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.239487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.239458    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:24:34.243994 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:34.243965    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58d5c75f99-mlzw8"]
Sep 11 17:24:35.686751 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:35.686719    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" path="/var/lib/kubelet/pods/5eba3ea3-4415-4fc6-ad22-3b3bed789114/volumes"
Sep 11 17:24:43.007740 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.007690    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" containerID="cri-o://8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf" gracePeriod=45
Sep 11 17:24:43.125079 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.125025    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 11 17:24:43.665595 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.665570    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:24:43.711226 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711194    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711390 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711262    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711390 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711304    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711390 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711333    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5flcn\" (UniqueName: \"kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711390 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711362    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711412    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711438    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711510    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm\") pod \"34e903b5-5096-47fb-adfc-dc2e231939aa\" (UID: \"34e903b5-5096-47fb-adfc-dc2e231939aa\") "
Sep 11 17:24:43.711615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711551    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:43.711860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711689    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:24:43.711937 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711916    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34e903b5-5096-47fb-adfc-dc2e231939aa-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.712003 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.711943    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.712323 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.712282    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home" (OuterVolumeSpecName: "home") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:43.714131 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.714000    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:43.714131 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.714092    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn" (OuterVolumeSpecName: "kube-api-access-5flcn") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "kube-api-access-5flcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:24:43.714376 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.714354    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm" (OuterVolumeSpecName: "dshm") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:43.724218 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.724195    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:43.734074 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.734051    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34e903b5-5096-47fb-adfc-dc2e231939aa" (UID: "34e903b5-5096-47fb-adfc-dc2e231939aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:43.812966 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812936    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.812966 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812961    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flcn\" (UniqueName: \"kubernetes.io/projected/34e903b5-5096-47fb-adfc-dc2e231939aa-kube-api-access-5flcn\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.813104 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812972    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.813104 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812983    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e903b5-5096-47fb-adfc-dc2e231939aa-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.813104 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812991    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.813104 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.812999    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e903b5-5096-47fb-adfc-dc2e231939aa-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:24:43.966087 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.965996    2550 generic.go:356] "Generic (PLEG): container finished" podID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerID="8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf" exitCode=0
Sep 11 17:24:43.966087 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.966053    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"
Sep 11 17:24:43.966290 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.966079    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerDied","Data":"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"}
Sep 11 17:24:43.966290 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.966138    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7" event={"ID":"34e903b5-5096-47fb-adfc-dc2e231939aa","Type":"ContainerDied","Data":"4957cf92fb1471e1c82b42fd85d3ef74188751bffc1342ffe7ac943c2d01e5d0"}
Sep 11 17:24:43.966290 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.966161    2550 scope.go:117] "RemoveContainer" containerID="8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"
Sep 11 17:24:43.975696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.975390    2550 scope.go:117] "RemoveContainer" containerID="264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"
Sep 11 17:24:43.985844 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.985823    2550 scope.go:117] "RemoveContainer" containerID="8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"
Sep 11 17:24:43.986122 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:24:43.986098    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf\": container with ID starting with 8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf not found: ID does not exist" containerID="8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"
Sep 11 17:24:43.986208 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.986134    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf"} err="failed to get container status \"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf\": rpc error: code = NotFound desc = could not find container \"8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf\": container with ID starting with 8e642decfddcac1428b56e5a3c460c74594bfbe5379dddfe215e305c961257cf not found: ID does not exist"
Sep 11 17:24:43.986208 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.986157    2550 scope.go:117] "RemoveContainer" containerID="264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"
Sep 11 17:24:43.986405 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:24:43.986384    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea\": container with ID starting with 264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea not found: ID does not exist" containerID="264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"
Sep 11 17:24:43.986445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.986412    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea"} err="failed to get container status \"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea\": rpc error: code = NotFound desc = could not find container \"264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea\": container with ID starting with 264da94c10d2a2d360eb299b0e599cf9d9994eee8861c2af0332f25ce39b00ea not found: ID does not exist"
Sep 11 17:24:43.990604 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.990580    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:24:43.996310 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:43.996288    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bfcdbf8bb-5szk7"]
Sep 11 17:24:45.686661 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:45.686608    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" path="/var/lib/kubelet/pods/34e903b5-5096-47fb-adfc-dc2e231939aa/volumes"
Sep 11 17:24:48.004652 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:48.004605    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:48.106328 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:48.106287    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:48.207686 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:48.207648    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:53.766618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:53.766579    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:53.880954 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:53.880920    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:54.403891 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:24:54.403851    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:08.352758 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:08.352723    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:08.445225 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:08.445195    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:08.617297 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:08.617206    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:08.718012 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:08.717984    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:11.440146 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:11.440098    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:14.426235 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:14.426199    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:19.906581 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:19.906545    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:19.974196 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:19.974166    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:31.696817 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.696779    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697121    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="storage-initializer"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697133    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="storage-initializer"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697143    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697149    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697162    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697167    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697179    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="storage-initializer"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697184    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="storage-initializer"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697233    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e903b5-5096-47fb-adfc-dc2e231939aa" containerName="main"
Sep 11 17:25:31.697367 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.697241    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eba3ea3-4415-4fc6-ad22-3b3bed789114" containerName="main"
Sep 11 17:25:31.700676 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.700651    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.703708 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.703680    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:25:31.703870 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.703848    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 11 17:25:31.703944 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.703860    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-nksll\""
Sep 11 17:25:31.703944 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.703928    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 11 17:25:31.704061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.703968    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 11 17:25:31.704061 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.704058    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 11 17:25:31.704873 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.704853    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4527t\""
Sep 11 17:25:31.721283 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.721259    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:25:31.749739 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.749711    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:31.824512 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824515    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824538    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824560    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824618 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824594    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glszl\" (UniqueName: \"kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824797 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824659    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824797 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824695    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.824797 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.824723    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925258 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925226    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925264    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925286    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925315    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glszl\" (UniqueName: \"kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925393 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925380    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925433    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925459    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925502    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925782 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925763    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925885 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925862    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925945 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925892    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.925992 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.925955    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.926036 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.926018    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.927636 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.927608    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.928029 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.928002    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:31.933303 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:31.933279    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glszl\" (UniqueName: \"kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl\") pod \"router-with-refs-pd-test-kserve-6dff5cc88f-xz25n\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:32.012066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:32.012000    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:32.097317 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:32.097287    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:32.195039 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:32.195002    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:32.289963 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:32.289934    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:25:32.291254 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:25:32.291221    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b76fabe_3109_4119_88a1_e07018ff9498.slice/crio-aacb0435ba4357c607c7234dfeeda967a443fa8346e018820cd208dca6ee6ade WatchSource:0}: Error finding container aacb0435ba4357c607c7234dfeeda967a443fa8346e018820cd208dca6ee6ade: Status 404 returned error can't find the container with id aacb0435ba4357c607c7234dfeeda967a443fa8346e018820cd208dca6ee6ade
Sep 11 17:25:33.195237 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:33.195202    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerStarted","Data":"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"}
Sep 11 17:25:33.195665 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:33.195254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerStarted","Data":"aacb0435ba4357c607c7234dfeeda967a443fa8346e018820cd208dca6ee6ade"}
Sep 11 17:25:33.195665 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:33.195313    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:34.201794 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:34.201758    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerStarted","Data":"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"}
Sep 11 17:25:36.214763 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:36.214730    2550 generic.go:356] "Generic (PLEG): container finished" podID="5b76fabe-3109-4119-88a1-e07018ff9498" containerID="ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a" exitCode=0
Sep 11 17:25:36.215169 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:36.214804    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerDied","Data":"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"}
Sep 11 17:25:37.221502 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:37.221463    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerStarted","Data":"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"}
Sep 11 17:25:37.248736 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:37.248688    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podStartSLOduration=4.077585432 podStartE2EDuration="6.248672554s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.171087122s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:37.246611711 +0000 UTC m=+4128.021312294" watchObservedRunningTime="2026-09-11 17:25:37.248672554 +0000 UTC m=+4128.023373131"
Sep 11 17:25:40.915040 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:40.915004    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:40.966804 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:40.966771    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:42.012449 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:42.012409    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:42.012990 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:42.012490    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:42.013776 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:42.013742    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:25:42.031030 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:42.031003    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:25:52.013051 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:25:52.012987    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:02.012570 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:02.012513    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:12.013343 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:12.013282    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:22.013355 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:22.013308    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:32.012928 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:32.012876    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:42.012437 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:42.012390    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:26:44.404528 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:44.404473    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:52.012515 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:26:52.012463    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:27:02.013200 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:02.013150    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:27:04.315563 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:04.315525    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:04.414952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:04.414913    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:10.148843 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:10.148808    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:10.249580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:10.249538    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:12.012524 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:12.012480    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:27:15.439385 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:15.439327    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:15.719687 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:15.719655    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:22.013318 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:22.013265    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8001/health\": dial tcp 10.132.0.67:8001: connect: connection refused"
Sep 11 17:27:26.667251 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:26.667208    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:32.028237 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:32.028202    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:27:32.040023 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:32.039998    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:27:33.382568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:33.382528    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:40.042464 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:40.042420    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:40.095796 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:40.095764    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:45.546588 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:45.546546    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:45.567804 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:45.567772    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:27:45.608553 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:27:45.608515    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:00.590778 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:00.590742    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main" containerID="cri-o://1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30" gracePeriod=45
Sep 11 17:28:01.206511 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.206450    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="llm-d-routing-sidecar" containerID="cri-o://4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b" gracePeriod=45
Sep 11 17:28:01.447485 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.447459    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:28:01.606653 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606609    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606674    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606701    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glszl\" (UniqueName: \"kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606732    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606764    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606783    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606838    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606860    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home\") pod \"5b76fabe-3109-4119-88a1-e07018ff9498\" (UID: \"5b76fabe-3109-4119-88a1-e07018ff9498\") "
Sep 11 17:28:01.607114 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.606924    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.607558 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.607197    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.607558 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.607208    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:28:01.608186 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.608160    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home" (OuterVolumeSpecName: "home") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.608954 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.608931    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm" (OuterVolumeSpecName: "dshm") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.609119 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.609092    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl" (OuterVolumeSpecName: "kube-api-access-glszl") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "kube-api-access-glszl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:01.609525 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.609500    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:28:01.621090 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.621063    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.630566 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.630544    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b76fabe-3109-4119-88a1-e07018ff9498" (UID: "5b76fabe-3109-4119-88a1-e07018ff9498"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.708534 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708510    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708534 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708532    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glszl\" (UniqueName: \"kubernetes.io/projected/5b76fabe-3109-4119-88a1-e07018ff9498-kube-api-access-glszl\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708702 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708544    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b76fabe-3109-4119-88a1-e07018ff9498-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708702 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708555    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708702 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708563    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708702 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708570    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b76fabe-3109-4119-88a1-e07018ff9498-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.708702 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.708578    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b76fabe-3109-4119-88a1-e07018ff9498-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.949927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.949856    2550 generic.go:356] "Generic (PLEG): container finished" podID="5b76fabe-3109-4119-88a1-e07018ff9498" containerID="1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30" exitCode=0
Sep 11 17:28:01.949927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.949879    2550 generic.go:356] "Generic (PLEG): container finished" podID="5b76fabe-3109-4119-88a1-e07018ff9498" containerID="4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b" exitCode=0
Sep 11 17:28:01.949927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.949914    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"
Sep 11 17:28:01.950171 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.949924    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerDied","Data":"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"}
Sep 11 17:28:01.950171 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.949979    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerDied","Data":"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"}
Sep 11 17:28:01.950171 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.950005    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n" event={"ID":"5b76fabe-3109-4119-88a1-e07018ff9498","Type":"ContainerDied","Data":"aacb0435ba4357c607c7234dfeeda967a443fa8346e018820cd208dca6ee6ade"}
Sep 11 17:28:01.950171 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.950032    2550 scope.go:117] "RemoveContainer" containerID="1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"
Sep 11 17:28:01.958656 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.958639    2550 scope.go:117] "RemoveContainer" containerID="ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"
Sep 11 17:28:01.969456 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.969434    2550 scope.go:117] "RemoveContainer" containerID="4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"
Sep 11 17:28:01.970755 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.970737    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:28:01.974144 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.974125    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6dff5cc88f-xz25n"]
Sep 11 17:28:01.977747 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.977727    2550 scope.go:117] "RemoveContainer" containerID="1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"
Sep 11 17:28:01.977994 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:28:01.977975    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30\": container with ID starting with 1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30 not found: ID does not exist" containerID="1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"
Sep 11 17:28:01.978063 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978002    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"} err="failed to get container status \"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30\": rpc error: code = NotFound desc = could not find container \"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30\": container with ID starting with 1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30 not found: ID does not exist"
Sep 11 17:28:01.978063 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978020    2550 scope.go:117] "RemoveContainer" containerID="ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"
Sep 11 17:28:01.978257 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:28:01.978244    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a\": container with ID starting with ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a not found: ID does not exist" containerID="ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"
Sep 11 17:28:01.978302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978259    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"} err="failed to get container status \"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a\": rpc error: code = NotFound desc = could not find container \"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a\": container with ID starting with ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a not found: ID does not exist"
Sep 11 17:28:01.978302 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978272    2550 scope.go:117] "RemoveContainer" containerID="4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"
Sep 11 17:28:01.978505 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:28:01.978492    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b\": container with ID starting with 4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b not found: ID does not exist" containerID="4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"
Sep 11 17:28:01.978557 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978507    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"} err="failed to get container status \"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b\": rpc error: code = NotFound desc = could not find container \"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b\": container with ID starting with 4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b not found: ID does not exist"
Sep 11 17:28:01.978557 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978517    2550 scope.go:117] "RemoveContainer" containerID="1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"
Sep 11 17:28:01.978752 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978735    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30"} err="failed to get container status \"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30\": rpc error: code = NotFound desc = could not find container \"1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30\": container with ID starting with 1135b2c6c35886b99fc36f45d1f0760abbd492ab3a014032141771c1049a0d30 not found: ID does not exist"
Sep 11 17:28:01.978752 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978750    2550 scope.go:117] "RemoveContainer" containerID="ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"
Sep 11 17:28:01.978987 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978971    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a"} err="failed to get container status \"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a\": rpc error: code = NotFound desc = could not find container \"ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a\": container with ID starting with ac623743aad8ffd94b8c90ca8111b18bed9636dab2a5bd1f978b14a0efcb484a not found: ID does not exist"
Sep 11 17:28:01.979039 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.978987    2550 scope.go:117] "RemoveContainer" containerID="4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"
Sep 11 17:28:01.979179 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:01.979164    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b"} err="failed to get container status \"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b\": rpc error: code = NotFound desc = could not find container \"4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b\": container with ID starting with 4e238edc5c70e81a882a360e4cac1530d9b7927d4e19f0f7f9c4cd11c67a805b not found: ID does not exist"
Sep 11 17:28:03.686851 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:03.686816    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" path="/var/lib/kubelet/pods/5b76fabe-3109-4119-88a1-e07018ff9498/volumes"
Sep 11 17:28:09.895372 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:09.895333    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:21.630498 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:21.630458    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:35.716067 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:35.716031    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:35.814236 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:35.814192    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:40.763485 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:40.763443    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:42.492448 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492359    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492917    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main"
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492936    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main"
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492949    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="llm-d-routing-sidecar"
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492958    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="llm-d-routing-sidecar"
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492970    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="storage-initializer"
Sep 11 17:28:42.493056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.492980    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="storage-initializer"
Sep 11 17:28:42.493384 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.493059    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="llm-d-routing-sidecar"
Sep 11 17:28:42.493384 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.493074    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b76fabe-3109-4119-88a1-e07018ff9498" containerName="main"
Sep 11 17:28:42.495281 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.495262    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.497928 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.497909    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 11 17:28:42.498213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.498191    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 11 17:28:42.498309 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.498196    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-wcpwz\""
Sep 11 17:28:42.498309 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.498284    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 11 17:28:42.498309 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.498285    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:28:42.498516 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.498497    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 11 17:28:42.508713 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.508689    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:28:42.518762 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.518740    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:42.649633 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hpcf\" (UniqueName: \"kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649777 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649680    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649777 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649859 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649818    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649899    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.649965 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.650002 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.649977    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751045 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.750974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hpcf\" (UniqueName: \"kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751045 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751008    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751185 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751134    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751185 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751164    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751268 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751201    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751268 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751253    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751290    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751337    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751351    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751609 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751585    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751711 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751647    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751711 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751697    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.751833 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.751814    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.753554 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.753534    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.753898 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.753880    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.763074 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.763048    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hpcf\" (UniqueName: \"kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf\") pod \"prestop-hook-test-kserve-b878d667c-gklxp\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.805578 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.805549    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:42.815130 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:42.815105    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:43.083296 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:43.083261    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:28:43.083934 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:28:43.083905    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd140a9f0_9dda_459b_97de_0eeab54f7a46.slice/crio-3fa68720fd9b62b51f2317d2d5a24ea268d92db1904e4d1cd59bb6f3105140e5 WatchSource:0}: Error finding container 3fa68720fd9b62b51f2317d2d5a24ea268d92db1904e4d1cd59bb6f3105140e5: Status 404 returned error can't find the container with id 3fa68720fd9b62b51f2317d2d5a24ea268d92db1904e4d1cd59bb6f3105140e5
Sep 11 17:28:43.142427 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:43.142397    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerStarted","Data":"3fa68720fd9b62b51f2317d2d5a24ea268d92db1904e4d1cd59bb6f3105140e5"}
Sep 11 17:28:44.150709 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:44.150657    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerStarted","Data":"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"}
Sep 11 17:28:46.164152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:46.164114    2550 generic.go:356] "Generic (PLEG): container finished" podID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerID="85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933" exitCode=0
Sep 11 17:28:46.164152 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:46.164148    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerDied","Data":"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"}
Sep 11 17:28:47.171038 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:47.171006    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerStarted","Data":"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"}
Sep 11 17:28:47.197400 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:47.197338    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podStartSLOduration=2.988510756 podStartE2EDuration="5.19732267s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.208811914s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:47.194206183 +0000 UTC m=+4317.968906773" watchObservedRunningTime="2026-09-11 17:28:47.19732267 +0000 UTC m=+4317.972023248"
Sep 11 17:28:47.806020 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:47.805986    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:52.806172 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:52.806132    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:28:52.807479 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:52.807454    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:28:53.988437 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:28:53.988391    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:01.806860 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:01.806821    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:02.806257 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:02.806215    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:29:02.963826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:02.963795    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:08.521596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:08.521557    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:12.806175 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:12.806129    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:29:20.322458 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:20.322419    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:20.432781 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:20.432743    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:22.806596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:22.806549    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:29:32.806344 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:32.806299    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:29:33.612263 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:33.612233    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:39.512572 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:39.512532    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:39.581592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:39.581529    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.806895 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:42.806848    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:29:52.806164 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:29:52.806115    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:30:02.806546 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:02.806504    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:30:10.438617 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:10.438573    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:10.534613 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:10.534572    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:12.806480 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:12.806385    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:30:16.490949 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:16.490910    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:21.708285 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:21.708240    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:21.949908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:21.949869    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:22.455703 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:22.455669    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:22.555159 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:22.555122    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:22.806380 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:22.806286    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:30:32.806949 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:32.806899    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:30:39.558146 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:39.558104    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:42.816960 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:42.816928    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:30:42.824730 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:42.824701    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:30:46.503927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.503882    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:30:46.530088 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.530055    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:30:46.533454 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.533427    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.547512 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.547488    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:30:46.683376 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683345    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlptf\" (UniqueName: \"kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683399    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683455    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683478    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683508    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683580 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683565    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683856 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683603    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.683856 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.683660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.784649 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.784649 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.784649 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784612    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.784885 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlptf\" (UniqueName: \"kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.784945 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785006 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.784961    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785063 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785124 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785066    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785124 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785086    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785231 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785022    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785336 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785307    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785467 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785428    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.785827 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.785807    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.787066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.787043    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.787281 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.787259    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.792951 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.792932    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlptf\" (UniqueName: \"kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf\") pod \"prestop-hook-test-kserve-b878d667c-x9cjw\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:46.843669 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:46.843648    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:47.119403 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:47.119374    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:30:47.121057 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:30:47.121028    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda688b7df_6ac4_42c8_95e6_7f5e6bf839d1.slice/crio-254956605ab2aa8ee1503fd92f235b70e0e9cc8dfba4670011f8e78a495276cd WatchSource:0}: Error finding container 254956605ab2aa8ee1503fd92f235b70e0e9cc8dfba4670011f8e78a495276cd: Status 404 returned error can't find the container with id 254956605ab2aa8ee1503fd92f235b70e0e9cc8dfba4670011f8e78a495276cd
Sep 11 17:30:47.780133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:47.780091    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerStarted","Data":"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"}
Sep 11 17:30:47.780546 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:47.780161    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerStarted","Data":"254956605ab2aa8ee1503fd92f235b70e0e9cc8dfba4670011f8e78a495276cd"}
Sep 11 17:30:49.791734 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:49.791698    2550 generic.go:356] "Generic (PLEG): container finished" podID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerID="44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b" exitCode=0
Sep 11 17:30:49.792217 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:49.791778    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerDied","Data":"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"}
Sep 11 17:30:50.799518 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:50.799471    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerStarted","Data":"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"}
Sep 11 17:30:50.824371 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:50.824322    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" podStartSLOduration=2.716837695 podStartE2EDuration="4.824306187s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.107468492s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:50.821312684 +0000 UTC m=+4441.596013286" watchObservedRunningTime="2026-09-11 17:30:50.824306187 +0000 UTC m=+4441.599006814"
Sep 11 17:30:50.844388 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:50.844364    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:56.844493 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:56.844453    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:30:56.846223 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:30:56.846186    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 11 17:31:01.526315 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:01.526264    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" containerID="cri-o://89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c" gracePeriod=45
Sep 11 17:31:01.817956 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:01.817853    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 11 17:31:02.198179 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.198153    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:31:02.235766 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235724    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.235952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235779    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.235952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235837    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.235952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235871    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.235952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235898    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.235952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235938    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.236256 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.235983    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.236256 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.236046    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hpcf\" (UniqueName: \"kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf\") pod \"d140a9f0-9dda-459b-97de-0eeab54f7a46\" (UID: \"d140a9f0-9dda-459b-97de-0eeab54f7a46\") "
Sep 11 17:31:02.236256 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.236237    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache" (OuterVolumeSpecName: "model-cache") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:02.236418 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.236374    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.236478 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.236412    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home" (OuterVolumeSpecName: "home") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:02.238356 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.236657    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:02.240362 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.240163    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm" (OuterVolumeSpecName: "dshm") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:02.240362 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.240227    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:02.240687 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.240656    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf" (OuterVolumeSpecName: "kube-api-access-8hpcf") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "kube-api-access-8hpcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:02.253787 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.253745    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:02.264273 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.264244    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d140a9f0-9dda-459b-97de-0eeab54f7a46" (UID: "d140a9f0-9dda-459b-97de-0eeab54f7a46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:02.336849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336814    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d140a9f0-9dda-459b-97de-0eeab54f7a46-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.336849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336843    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.336849 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336855    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.337066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336865    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hpcf\" (UniqueName: \"kubernetes.io/projected/d140a9f0-9dda-459b-97de-0eeab54f7a46-kube-api-access-8hpcf\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.337066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336873    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d140a9f0-9dda-459b-97de-0eeab54f7a46-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.337066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336881    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.337066 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.336888    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d140a9f0-9dda-459b-97de-0eeab54f7a46-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:02.865865 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.865833    2550 generic.go:356] "Generic (PLEG): container finished" podID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerID="89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c" exitCode=0
Sep 11 17:31:02.866287 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.865888    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"
Sep 11 17:31:02.866287 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.865891    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerDied","Data":"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"}
Sep 11 17:31:02.866287 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.866007    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp" event={"ID":"d140a9f0-9dda-459b-97de-0eeab54f7a46","Type":"ContainerDied","Data":"3fa68720fd9b62b51f2317d2d5a24ea268d92db1904e4d1cd59bb6f3105140e5"}
Sep 11 17:31:02.866287 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.866033    2550 scope.go:117] "RemoveContainer" containerID="89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"
Sep 11 17:31:02.880319 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.880283    2550 scope.go:117] "RemoveContainer" containerID="85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"
Sep 11 17:31:02.890487 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.890468    2550 scope.go:117] "RemoveContainer" containerID="89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"
Sep 11 17:31:02.890808 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:31:02.890783    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c\": container with ID starting with 89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c not found: ID does not exist" containerID="89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"
Sep 11 17:31:02.890927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.890814    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c"} err="failed to get container status \"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c\": rpc error: code = NotFound desc = could not find container \"89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c\": container with ID starting with 89a689803316b2c75566ca05d904a96672611f996baa2b15ad313edc4e888b4c not found: ID does not exist"
Sep 11 17:31:02.890927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.890835    2550 scope.go:117] "RemoveContainer" containerID="85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"
Sep 11 17:31:02.891133 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:31:02.891112    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933\": container with ID starting with 85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933 not found: ID does not exist" containerID="85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"
Sep 11 17:31:02.891176 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.891140    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933"} err="failed to get container status \"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933\": rpc error: code = NotFound desc = could not find container \"85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933\": container with ID starting with 85e0f614e681e9046bd9e71685f33e01d70849c51b4e13f6cfdf1435ba083933 not found: ID does not exist"
Sep 11 17:31:02.895973 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.895950    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:31:02.898898 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:02.898874    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-gklxp"]
Sep 11 17:31:03.686639 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:03.686593    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" path="/var/lib/kubelet/pods/d140a9f0-9dda-459b-97de-0eeab54f7a46/volumes"
Sep 11 17:31:06.844200 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:06.844140    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 11 17:31:16.844686 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:16.844604    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 11 17:31:21.026855 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:21.026820    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:31:21.034771 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:21.034741    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:21.105848 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:21.105812    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:21.735948 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:21.735911    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:21.948451 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:21.948414    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:36.050040 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:36.049979    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main" containerID="cri-o://385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9" gracePeriod=45
Sep 11 17:31:37.788350 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.788327    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-b878d667c-x9cjw_a688b7df-6ac4-42c8-95e6-7f5e6bf839d1/main/0.log"
Sep 11 17:31:37.790992 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.790974    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:31:37.880422 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880339    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlptf\" (UniqueName: \"kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880422 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880399    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880428    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880453    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880474    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880509    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.880662 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880572    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home\") pod \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\" (UID: \"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1\") "
Sep 11 17:31:37.881028 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880778    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:37.881028 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.880795    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:37.881133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.881035    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tmp-dir\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.881133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.881057    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-model-cache\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.881133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.881057    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:31:37.881133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.881070    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home" (OuterVolumeSpecName: "home") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:37.882837 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.882815    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:37.882837 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.882812    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm" (OuterVolumeSpecName: "dshm") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:37.882988 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.882924    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf" (OuterVolumeSpecName: "kube-api-access-tlptf") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "kube-api-access-tlptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:37.900203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.900178    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" (UID: "a688b7df-6ac4-42c8-95e6-7f5e6bf839d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:37.982419 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982389    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlptf\" (UniqueName: \"kubernetes.io/projected/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kube-api-access-tlptf\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.982419 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982415    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-dshm\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.982582 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982424    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-kserve-provision-location\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.982582 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982434    2550 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-cabundle-cert\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.982582 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982443    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-tls-certs\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:37.982582 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:37.982450    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1-home\") on node \"ip-10-0-135-175.ec2.internal\" DevicePath \"\""
Sep 11 17:31:38.057819 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.057795    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-b878d667c-x9cjw_a688b7df-6ac4-42c8-95e6-7f5e6bf839d1/main/0.log"
Sep 11 17:31:38.058235 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.058210    2550 generic.go:356] "Generic (PLEG): container finished" podID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerID="385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9" exitCode=130
Sep 11 17:31:38.058338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.058262    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerDied","Data":"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"}
Sep 11 17:31:38.058338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.058272    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"
Sep 11 17:31:38.058338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.058307    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw" event={"ID":"a688b7df-6ac4-42c8-95e6-7f5e6bf839d1","Type":"ContainerDied","Data":"254956605ab2aa8ee1503fd92f235b70e0e9cc8dfba4670011f8e78a495276cd"}
Sep 11 17:31:38.058338 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.058329    2550 scope.go:117] "RemoveContainer" containerID="385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"
Sep 11 17:31:38.068157 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.068141    2550 scope.go:117] "RemoveContainer" containerID="44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"
Sep 11 17:31:38.075739 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.075723    2550 scope.go:117] "RemoveContainer" containerID="385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"
Sep 11 17:31:38.076012 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:31:38.075993    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9\": container with ID starting with 385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9 not found: ID does not exist" containerID="385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"
Sep 11 17:31:38.076098 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.076024    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9"} err="failed to get container status \"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9\": rpc error: code = NotFound desc = could not find container \"385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9\": container with ID starting with 385ef721ba6ef0f9d688170a298f48f9b75d45dbe2a12fc0ac660afc0826ace9 not found: ID does not exist"
Sep 11 17:31:38.076098 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.076047    2550 scope.go:117] "RemoveContainer" containerID="44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"
Sep 11 17:31:38.076336 ip-10-0-135-175 kubenswrapper[2550]: E0911 17:31:38.076319    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b\": container with ID starting with 44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b not found: ID does not exist" containerID="44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"
Sep 11 17:31:38.076377 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.076341    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b"} err="failed to get container status \"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b\": rpc error: code = NotFound desc = could not find container \"44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b\": container with ID starting with 44cf51219e2e553fcb0564694ad0e72db24497f82a3e660e1ba62cd9a640064b not found: ID does not exist"
Sep 11 17:31:38.081352 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.081324    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:31:38.086666 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:38.086644    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-b878d667c-x9cjw"]
Sep 11 17:31:39.686677 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:31:39.686642    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" path="/var/lib/kubelet/pods/a688b7df-6ac4-42c8-95e6-7f5e6bf839d1/volumes"
Sep 11 17:32:09.097553 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:09.097514    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:10.708210 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:10.708172    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:13.643202 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:13.643150    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:13.737870 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:13.737840    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:15.363529 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:15.363467    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:16.105975 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:16.105940    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:20.953430 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:20.953385    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:21.008325 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:21.008292    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:21.612420 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:21.612391    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:21.744325 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:21.744284    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:35.761108 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:35.761074    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:32:35.789263 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:35.789232    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:32:36.506091 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:36.506053    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:32:36.532850 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:36.532817    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:32:37.297144 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:37.297113    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:32:37.326669 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:37.326604    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:32:38.015826 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:38.015794    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:32:38.044691 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:38.044655    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:32:39.634325 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:39.634287    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:39.641885 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:39.641858    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:41.486239 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:41.486205    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:41.543566 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:41.543531    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:43.716927 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:43.716887    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:43.781388 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:43.781358    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:45.413149 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:45.413121    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:45.428522 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:45.428495    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:47.067472 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:47.067441    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:47.079884 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:47.079856    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:48.699964 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:48.699924    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:48.712084 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:48.712056    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:50.356133 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:50.356082    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:50.367870 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:50.367843    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:52.024809 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:52.024770    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:52.037084 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:52.037056    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:53.659325 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:53.659288    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:53.671207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:53.671182    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:55.298980 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:55.298940    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:55.315538 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:55.315511    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:56.958656 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:56.958603    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:56.970940 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:56.970913    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:32:58.618082 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:58.618045    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:32:58.629730 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:32:58.629705    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:00.252899 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:00.252863    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:00.265362 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:00.265337    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:01.913466 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:01.913427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:01.925241 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:01.925218    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:03.581723 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:03.581689    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:03.593589 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:03.593561    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:05.226681 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:05.226646    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:05.238442 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:05.238416    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:06.855814 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:06.855776    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:06.867345 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:06.867317    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:08.488536 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:08.488495    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:08.499914 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:08.499890    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:10.045760 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:10.045730    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:10.124717 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:10.124674    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:11.739886 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:11.739851    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:11.754026 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:11.754003    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:12.413615 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:12.413524    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:12.476292 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:12.476252    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:13.376518 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:13.376483    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:13.390366 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:13.390339    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:15.029715 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:15.029675    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:15.040828 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:15.040801    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:16.660465 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:16.660427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:16.672088 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:16.672060    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:18.302419 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:18.302385    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:18.314553 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:18.314529    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:19.939584 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:19.939548    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:19.951108 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:19.951080    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:21.916910 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:21.916874    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:22.080384 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:22.080341    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:22.177587 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:22.177477    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.121712 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:25.121676    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6wc5s_9a41d8fe-c043-4cf0-b634-afbb8447d967/global-pull-secret-syncer/0.log"
Sep 11 17:33:25.241562 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:25.241532    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log"
Sep 11 17:33:25.398596 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:25.398516    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-175.ec2.internal_a31c9521de76f63a6f298dd8cdc84d1f/haproxy/0.log"
Sep 11 17:33:27.809484 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:27.809447    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:27.879213 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:27.879183    2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qbgc_a9e08126-0dfc-43dc-a899-0597def90b74/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:31.022550 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:31.022517    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-dww5t_4060dd60-2fee-4751-9a71-0397b53311c2/manager/0.log"
Sep 11 17:33:31.044303 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:31.044269    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8vgwp_e0d771fc-bbb3-4ae9-88de-4f3b5a731810/limitador/0.log"
Sep 11 17:33:32.205269 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.205236    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/alertmanager/0.log"
Sep 11 17:33:32.227163 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.227140    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/config-reloader/0.log"
Sep 11 17:33:32.246568 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.246545    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/kube-rbac-proxy-web/0.log"
Sep 11 17:33:32.267274 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.267254    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/kube-rbac-proxy/0.log"
Sep 11 17:33:32.289952 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.289895    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/kube-rbac-proxy-metric/0.log"
Sep 11 17:33:32.313409 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.313381    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/prom-label-proxy/0.log"
Sep 11 17:33:32.337013 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.336989    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b2b9674-3a7a-46e8-a189-eef89f8defc7/init-config-reloader/0.log"
Sep 11 17:33:32.431606 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.431579    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4qsjw_5074fd5a-de72-47a6-b80c-48480094d971/kube-state-metrics/0.log"
Sep 11 17:33:32.452671 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.452645    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4qsjw_5074fd5a-de72-47a6-b80c-48480094d971/kube-rbac-proxy-main/0.log"
Sep 11 17:33:32.478443 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.478424    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4qsjw_5074fd5a-de72-47a6-b80c-48480094d971/kube-rbac-proxy-self/0.log"
Sep 11 17:33:32.506520 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.506495    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5949df56d6-gvqpx_9accd2e3-d2ee-403c-a246-8684f7c01044/metrics-server/0.log"
Sep 11 17:33:32.753361 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.753329    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwnps_256e8b72-ac3e-4292-9484-57f0c8e03b47/node-exporter/0.log"
Sep 11 17:33:32.772592 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.772571    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwnps_256e8b72-ac3e-4292-9484-57f0c8e03b47/kube-rbac-proxy/0.log"
Sep 11 17:33:32.792397 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.792375    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nwnps_256e8b72-ac3e-4292-9484-57f0c8e03b47/init-textfile/0.log"
Sep 11 17:33:32.819131 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.819110    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-28hm5_0c27878f-25c4-4010-9cd9-a4c38bb1b394/kube-rbac-proxy-main/0.log"
Sep 11 17:33:32.839930 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.839908    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-28hm5_0c27878f-25c4-4010-9cd9-a4c38bb1b394/kube-rbac-proxy-self/0.log"
Sep 11 17:33:32.861398 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.861374    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-28hm5_0c27878f-25c4-4010-9cd9-a4c38bb1b394/openshift-state-metrics/0.log"
Sep 11 17:33:32.906147 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.906124    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/prometheus/0.log"
Sep 11 17:33:32.922462 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.922446    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/config-reloader/0.log"
Sep 11 17:33:32.943550 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.943529    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/thanos-sidecar/0.log"
Sep 11 17:33:32.964207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.964179    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/kube-rbac-proxy-web/0.log"
Sep 11 17:33:32.986460 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:32.986437    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/kube-rbac-proxy/0.log"
Sep 11 17:33:33.007696 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:33.007612    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/kube-rbac-proxy-thanos/0.log"
Sep 11 17:33:33.029276 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:33.029259    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_37f7e6ae-9bef-4137-96d2-a36198da4d39/init-config-reloader/0.log"
Sep 11 17:33:33.152374 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:33.152343    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684c788d95-h7dzh_610aabb3-ea79-426e-afee-9a10c429506a/telemeter-client/0.log"
Sep 11 17:33:33.173506 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:33.173484    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684c788d95-h7dzh_610aabb3-ea79-426e-afee-9a10c429506a/reload/0.log"
Sep 11 17:33:33.195021 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:33.195002    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684c788d95-h7dzh_610aabb3-ea79-426e-afee-9a10c429506a/kube-rbac-proxy/0.log"
Sep 11 17:33:35.129050 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129009    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"]
Sep 11 17:33:35.129542 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129520    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="storage-initializer"
Sep 11 17:33:35.129603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129558    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="storage-initializer"
Sep 11 17:33:35.129603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129569    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="storage-initializer"
Sep 11 17:33:35.129603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129576    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="storage-initializer"
Sep 11 17:33:35.129603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129591    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main"
Sep 11 17:33:35.129603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129598    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main"
Sep 11 17:33:35.129813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129636    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main"
Sep 11 17:33:35.129813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129642    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main"
Sep 11 17:33:35.129813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129699    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d140a9f0-9dda-459b-97de-0eeab54f7a46" containerName="main"
Sep 11 17:33:35.129813 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.129710    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a688b7df-6ac4-42c8-95e6-7f5e6bf839d1" containerName="main"
Sep 11 17:33:35.132953 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.132937    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.135768 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.135746    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"openshift-service-ca.crt\""
Sep 11 17:33:35.136899 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.136873    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvg9x\"/\"default-dockercfg-5vsk4\""
Sep 11 17:33:35.136995 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.136909    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"kube-root-ca.crt\""
Sep 11 17:33:35.144234 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.144212    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"]
Sep 11 17:33:35.262964 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.262930    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-proc\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.262964 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.262964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-podres\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.263193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.262994    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-sys\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.263193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.263076    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drktg\" (UniqueName: \"kubernetes.io/projected/14e69944-b1f9-4c91-9062-7528bc23498c-kube-api-access-drktg\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.263193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.263112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-lib-modules\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.363764 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363733    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-proc\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.363908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363771    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-podres\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.363908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-sys\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.363908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363856    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-proc\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.363908 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363863    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drktg\" (UniqueName: \"kubernetes.io/projected/14e69944-b1f9-4c91-9062-7528bc23498c-kube-api-access-drktg\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.364069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363908    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-podres\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.364069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-lib-modules\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.364069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363933    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-sys\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.364069 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.363993    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14e69944-b1f9-4c91-9062-7528bc23498c-lib-modules\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.371562 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.371537    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drktg\" (UniqueName: \"kubernetes.io/projected/14e69944-b1f9-4c91-9062-7528bc23498c-kube-api-access-drktg\") pod \"perf-node-gather-daemonset-gxg9w\" (UID: \"14e69944-b1f9-4c91-9062-7528bc23498c\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.443203 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.443129    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:35.713331 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:35.713307    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"]
Sep 11 17:33:35.715021 ip-10-0-135-175 kubenswrapper[2550]: W0911 17:33:35.714995    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14e69944_b1f9_4c91_9062_7528bc23498c.slice/crio-0c0c3745931610b69941f1bc1e19c5c02fa792bdf116ad8536027e1cb38de854 WatchSource:0}: Error finding container 0c0c3745931610b69941f1bc1e19c5c02fa792bdf116ad8536027e1cb38de854: Status 404 returned error can't find the container with id 0c0c3745931610b69941f1bc1e19c5c02fa792bdf116ad8536027e1cb38de854
Sep 11 17:33:36.024207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.024130    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-dsqt5_f2947302-28e5-4bfd-871c-0b402161474c/volume-data-source-validator/0.log"
Sep 11 17:33:36.619426 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.619385    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w" event={"ID":"14e69944-b1f9-4c91-9062-7528bc23498c","Type":"ContainerStarted","Data":"a9293325b90b9b32abc438e2acc7ee8f6df9df14fcd116a83fa14e172aaf26d3"}
Sep 11 17:33:36.619943 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.619438    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w" event={"ID":"14e69944-b1f9-4c91-9062-7528bc23498c","Type":"ContainerStarted","Data":"0c0c3745931610b69941f1bc1e19c5c02fa792bdf116ad8536027e1cb38de854"}
Sep 11 17:33:36.619943 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.619474    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:36.638413 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.638365    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w" podStartSLOduration=1.6383522579999998 podStartE2EDuration="1.638352258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:36.635959742 +0000 UTC m=+4607.410660319" watchObservedRunningTime="2026-09-11 17:33:36.638352258 +0000 UTC m=+4607.413052835"
Sep 11 17:33:36.991675 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:36.991643    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2859b_7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663/dns/0.log"
Sep 11 17:33:37.011759 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:37.011727    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2859b_7de0b6f7-cfc7-4bb5-b33f-083ab3ce3663/kube-rbac-proxy/0.log"
Sep 11 17:33:37.124560 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:37.124520    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bfjtf_29a5160d-5b16-4a1c-aa25-d3e03ef9a013/dns-node-resolver/0.log"
Sep 11 17:33:37.794836 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:37.794806    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccx2w_7379ca9e-d28c-4754-926d-f13edc395a9c/node-ca/0.log"
Sep 11 17:33:38.858725 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:38.858691    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:33:38.886093 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:38.886065    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:33:39.436756 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:39.436724    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nnkfw_264a56a3-e3a2-4a4a-832d-2911100e725f/serve-healthcheck-canary/0.log"
Sep 11 17:33:40.004867 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:40.004840    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9b7c_374640ff-5896-484e-97f3-2de2abae07e7/kube-rbac-proxy/0.log"
Sep 11 17:33:40.023516 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:40.023485    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9b7c_374640ff-5896-484e-97f3-2de2abae07e7/exporter/0.log"
Sep 11 17:33:40.043325 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:40.043303    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b9b7c_374640ff-5896-484e-97f3-2de2abae07e7/extractor/0.log"
Sep 11 17:33:42.633445 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:42.633415    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-gxg9w"
Sep 11 17:33:43.007882 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:43.007854    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5956778578-6jkf6_2e39d478-1232-4dcf-9553-61b776e9708c/manager/0.log"
Sep 11 17:33:44.688300 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:44.688249    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:33:44.720944 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:44.720914    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:33:45.710207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:45.710122    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kt8qc_33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f/s3-init/0.log"
Sep 11 17:33:45.732780 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:45.732756    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kt8qc_33c4a3f0-0ae4-4dbc-8037-ead1f08c9f3f/download-hf-model/0.log"
Sep 11 17:33:46.014327 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:46.014236    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/0.log"
Sep 11 17:33:46.042058 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:46.042031    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-68dcb6f84c-svpft_83a68164-af95-4ca7-8e2c-1a55a7e2ce40/istio-proxy/1.log"
Sep 11 17:33:51.800193 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.800164    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/kube-multus-additional-cni-plugins/0.log"
Sep 11 17:33:51.819603 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.819579    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/egress-router-binary-copy/0.log"
Sep 11 17:33:51.838924 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.838904    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/cni-plugins/0.log"
Sep 11 17:33:51.859207 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.859185    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/bond-cni-plugin/0.log"
Sep 11 17:33:51.882547 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.882513    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/routeoverride-cni/0.log"
Sep 11 17:33:51.903070 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.903053    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/whereabouts-cni-bincopy/0.log"
Sep 11 17:33:51.922088 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:51.922069    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rgg4t_4c415c27-e7dd-4eae-a2ea-234ec2c85631/whereabouts-cni/0.log"
Sep 11 17:33:52.151566 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:52.151483    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dxhpb_e6234a20-e5b0-4098-9721-e0e098e2528b/kube-multus/0.log"
Sep 11 17:33:52.302280 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:52.302252    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wj4ht_c3698c03-eddd-48ee-882c-b8b7b3e9d9fc/network-metrics-daemon/0.log"
Sep 11 17:33:52.319056 ip-10-0-135-175 kubenswrapper[2550]: I0911 17:33:52.319025    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wj4ht_c3698c03-eddd-48ee-882c-b8b7b3e9d9fc/kube-rbac-proxy/0.log"